Sink configuration that collects to vectors.

struct FunctionalTable{C<:NamedTuple, O<:Tuple{Vararg{FunctionalTables.ColumnOrdering,N} where N}}

Internal notes

• Use accessors length, colums, and ordering to access the fields, property accessors are forwarded to columns.

• The only inner constructor is the one where both the length and the ordering is trusted (and thus unchecked). Outer constructors should first wrap the ordering rule, then compute/verify length.

FunctionalTable(itr)
FunctionalTable(itr, ordering_rule; cfg)

Create a FunctionalTable from an iterable that returns NamedTuples.

Returned values need to have the same names (but not necessarily types).

ordering_rule specifies sorting. The VerifyOrdering (default), TrustOrdering, and TryOrdering constructors take a tuple of a tuple of :key or :key => reverse elements.

cfg determines sink configuration for collecting elements of the columns, see SinkConfig.

RepeatRow(row)

A row repeated as many times as needed. Can be merged to a FunctionalTable, or instantiated with FunctionalTable(len, repeat_row).

struct SinkConfig{useRLE, M}

SinkConfig(; useRLE, missingvalue)

Make a sink configuration, using defaults.

Accept the specified ordering to hold without any checks (except for verifying that column names are valid).

Try the specified ordering, when necesary degrade to a subset of it that holds.

Verify that the specified ordering holds. This is the default ordering rule.

aggregator(f)

Wrap a function so that it maps columns of a FunctionalTable to a table with a single row, columwise, ignoring the index. Returns a closure.

Example

map(aggregator(mean), by(ft, :col))

will calculate means after grouping by :col.

by(ft, splitkeys)

An iterator that groups rows of tables by the columns splitkeys, returning (index::NamedTupe, table::FunctionalTable) for each contiguous block of the index keys.

The function has a convenience form by(ft, splitkeys...; ...).

columns(ft)

Return the columns in a NamedTuple.

Each column is an iterable, but not necessarily an <: AbstractVector.

ignoreindex(f)

Wrap a function returning a closure that ignores the first argument (the index in by mappings).

ordering(ft)

Return the ordering of the table, which is a tuple of ColumnOrdering objects.

picking(key)

rename(ft, changes)

Rename the columns of a FunctionalTable. The second argument should be a NamedTuple of src = dest pairs, where dest is a symbol.

Example

rename(ft, (α = :a, β = :b)) # rename `α` to `a` and `β` to `b`

wrapping(key)

Return a callable that wraps its argument in a NamedTuple with a given key.

Types accepted as valid column ordering specifications in the user API.

Shows this many values from each column in a FunctionalTable.

Default sink configuration.

ColumnOrdering(key)
ColumnOrdering(key, rev)

Process a column ordering specification, called by table_ordering.

struct ColumnOrdering{K, R}

Ordering specification for a column. K::Symbol is a key for ordering by isless, R::Bool == true reverses that for this key. Internal.

Type for keys, used internally.

struct OrderingRule{R, O<:Tuple{Vararg{FunctionalTables.ColumnOrdering,N} where N}}

Rule for dealing with specified orderings. Verifies key uniqueness.

See VerifyOrdering, TrustOrdering, and TryOrdering.

This type and its methods are internal.

struct RLEVector{C, T, S, anyS}

An RLE encoded vector, using negative lengths for missing values. Use the RLEVector{S}(C, T) constructor for creating an empty one.

When an elemenet in counts is positive, it encodes that many of the corresponding element in data.

Negative counts encode values of type S (has to be a concrete singleton). In this case there is no corresponding value in data, ie data may have fewer elements than counts. Note that 0 values in count are reserved, and currently should not happen.

The flag anyS::Bool is true iff there are any values of type S in object.

An RLEVector is iterable.

Create an empty RLEVector for Union{T,S}, with special-casing the singleton type S. RLE counts are stored in type C.

RepeatValue(value, len)

Equivalent to a vector containing len instances of value. Used internally.

struct SplitTable{K<:NamedTuple, B<:NamedTuple, O<:Tuple{Vararg{FunctionalTables.ColumnOrdering,N} where N}, T<:FunctionalTable}

Implements by using an iterator.

Internals

Each rows from the underlying FunctionalTable is split into index and rest.

Iterator state is

1. nothing when the rows of the underlying FunctionalTable have been exhausted,

2. index, rest, itrstate for the next block, where index and rest are the first

(mismatching) row that has not been pushed to the buffers.

Shorthand for table ordering. Internal.

struct TrustLength

Wrapper type to indicate that the length should not be checked.

filter(f, ft; cfg)

Filter a FunctionalTable using a prediate that maps rows (as NamedTuples) to Bool. Preserves type (and thus ordering) of the table.

first(ft, n; cfg)

A FunctionalTable of the first n rows.

Useful for previews and data exploration. Preserves type (and thus ordering) of the table.

getindex(ft, keep)

With a tuple of symbols returns FunctionalTable with a subset of the columns.

With a single symbol, return that column (an iterable).

[drop = spec] will keep all but the given columns, where spec is a Tuple of Symbols.

Example

julia ft[(:a, :b)] ft[:a] ft[drop = (:a, :b)]

map(f, st; cfg)

Map a table split with by using f.

Specifically, f(index, table) receives the split index (a NamedTuple) and a FunctionalTable.

It is supposed to return an iterable that returns rows (can be a FunctionalTable). These will be prepended with the corresponding index, and collected into a FunctionalTable with cfg.

When f returns just a single row (eg aggregation), wrap by Ref to create a single-element iterable.

map(f, ft; cfg)

merge(f, ft; cfg, replace)

Map ft using f by rows, then merge the two. See map(f, ::FunctionalTable).

cfg is passed to map, replace governs replacement of overlapping columns in merge.

merge(a, b; replace)

Merge two FunctionalTables.

When replace == true, columns in the first one are replaced by second one, otherwise an error is thrown if column names overlap.

The second table can be specified as a NamedTuple of columns.

append1(v, elt)

Append elt to v, allocating a new vector and copying the contents.

Type of new collection is calculated using promote_type.

cancontain(T, elt)

Test if a collection of element type T can contain a new element elt without any loss of precision.

cmp_ordering(ordering, a, b)

Compare a and b given the ordering.

Internal.

collect_column(cfg, itr)

Collect results from itr into a sink (using config cfg), then finalize and return the column.

len, columns, ordering_rule =

collect_columns(cfg, itr, ordering_rule)
collect_columns(cfg, itr, ordering_rule, known_types)

Collect results from itr, which are supposed to be NamedTuples with the same names, into sinks (using config cfg), then finalize and return

1. the length,

2. the NamedTuple of the columns, and

3. the ordering rule (which is always ::TrustOrdering, by construction).

Determine the names and types from the first named tuple, using known_types as the narrowest types for the given columns.

Special rules for empty iterators

When itr is empty, use a known_types will be used to create empty columns, and only the TryOrdering rule will be narrowed to these. Other rule with more column names may cause an error in the callee, which is intentional.

empty_sinks(cfg, ?)

Empty sinks for a named tuple of elements, using a type.

finalize_sink(?, sink)

Convert sink to a column.

sink may share structure with the result and is not supposed to be used for saving any more elements.

finalize_sinks(cfg, sinks)

Finalize a (named) tuple of sinks.

fuse(f, index, tables)

Prepend the index as repeated columns to f(index, tables...).

is_prefix(a, b)

Test if b starts with a.

key_in(key, keys)

Check if a key is in itr.

make_sink(cfg, ?)

Create and return a sink using configuration cfg that stores elements of type T. When T is unkown, use Base.Bottom.

mask_ordering(ordering, keys)
mask_ordering(ordering, keys, invert)

When invert == false, keep the initial part of ordering that has keys in keys. Not having a key in keys invalidates the tail ordering from that point. This is useful for selecting subsets of orderings.

When invert == true, having a key in keys invalidates the ordering. This is useful for orderings of merged and dropped columns.

mask_try_ordering(ordering_rule, keys)

For TryOrdering, return a masked ordering rule that is is defined on keys so that comparisons make sense, otherwise return the original ordering_rule (that will just error for undefined keys).

merge_default_types(rowtype, defaults)

Whenever defaults has a given key, use the corresponding type in rowtype, otherwise Union{}.

narrow(x)

Convert the argument to a narrower type if possible without losing precision.

!!! NOTE This function is not type stable, use only when new container types are determined.

ordering_repr(ordering)

A string representation of an ordering, eg for use in show.

Accessor for key. Internal.

retained_ordering(ordering, a, b)

Return the (initial) part of ordering under which a ≤ b.

split_compatible_ordering(ordering, splitkeys)

Extend ordering so that it can be used to split a table with splitkeys.

split_namedtuple(?, nt)

Splits a named tuple in two, based on the names in splitter.

Returns two NamedTuples; the first one is ordered as splitter, the second one with the remaining values as in the original argument.

julia> split_namedtuple(NamedTuple{(:a, :c)}, (c = 1, b = 2, a = 3, d = 4))
((a = 3, c = 1), (b = 2, d = 4))

split_table(ft, splitkeys)

Helper function to set up a SplitTable. Internal.

start_sinks(cfg, row, known_types)

Start sinks using row, using the default known_types when available.

store!_or_reallocate(?, sink, elt)

Either store elt in sink (in which case the returned value is ≡ sink), or allocate a new sink that can do this, copy the contents, save elt and return that (then the returned value is ≢ sink).

store!_or_reallocate_row(cfg, sinks, elts)

Broadcast store!_or_rellocate for a compatible (named) tuple of sinks and elts. Return the (potentially) new sinks.

table_ordering(column_ordering_specifications)

Process ordering specifications for columns (an iterable or possibly a TableOrdering), return a TableOrdering. Check for uniqueness (but not validity) of keys.

Accepted syntax:

• :key, for ordering a column in ascending order,

• :key => reverse, for ordering a column in descending order.

All functions which accept ordering specifications should use this, but the function itself is not part of the API.