struct FunctionalTable{C<:NamedTuple, O<:Tuple{Vararg{FunctionalTables.ColumnOrdering,N} where N}}

Internal notes

  • Use accessors length, colums, and ordering to access the fields, property accessors are forwarded to columns.

  • The only inner constructor is the one where both the length and the ordering is trusted (and thus unchecked). Outer constructors should first wrap the ordering rule, then compute/verify length.

FunctionalTable(itr, ordering_rule; cfg)

Create a FunctionalTable from an iterable that returns NamedTuples.

Returned values need to have the same names (but not necessarily types).

ordering_rule specifies sorting. The VerifyOrdering (default), TrustOrdering, and TryOrdering constructors take a tuple of a tuple of :key or :key => reverse elements.

cfg determines sink configuration for collecting elements of the columns, see SinkConfig.



A row repeated as many times as needed. Can be merged to a FunctionalTable, or instantiated with FunctionalTable(len, repeat_row).


Accept the specified ordering to hold without any checks (except for verifying that column names are valid).


This can lead to incorrect results, use cautiously. VerifyOrdering is recommended instead as it has little overhead.


Wrap a function so that it maps columns of a FunctionalTable to a table with a single row, columwise, ignoring the index. Returns a closure.


map(aggregator(mean), by(ft, :col))

will calculate means after grouping by :col.

by(ft, splitkeys)

An iterator that groups rows of tables by the columns splitkeys, returning (index::NamedTupe, table::FunctionalTable) for each contiguous block of the index keys.

The function has a convenience form by(ft, splitkeys...; ...).


Return the columns in a NamedTuple.

Each column is an iterable, but not necessarily an <: AbstractVector.


Never mutate columns obtained by this method, as that will violate invariants assumed by the implementation. Use map(collect, columns(ft)) or similar to obtain mutable vectors.


Return the ordering of the table, which is a tuple of ColumnOrdering objects.

rename(ft, changes)

Rename the columns of a FunctionalTable. The second argument should be a NamedTuple of src = dest pairs, where dest is a symbol.


rename(ft, (α = :a, β = :b)) # rename `α` to `a` and `β` to `b`

Return a callable that wraps its argument in a NamedTuple with a given key.

struct ColumnOrdering{K, R}

Ordering specification for a column. K::Symbol is a key for ordering by isless, R::Bool == true reverses that for this key. Internal.

struct RLEVector{C, T, S, anyS}

An RLE encoded vector, using negative lengths for missing values. Use the RLEVector{S}(C, T) constructor for creating an empty one.

When an elemenet in counts is positive, it encodes that many of the corresponding element in data.

Negative counts encode values of type S (has to be a concrete singleton). In this case there is no corresponding value in data, ie data may have fewer elements than counts. Note that 0 values in count are reserved, and currently should not happen.

The flag anyS::Bool is true iff there are any values of type S in object.

An RLEVector is iterable.


Create an empty RLEVector for Union{T,S}, with special-casing the singleton type S. RLE counts are stored in type C.

struct SplitTable{K<:NamedTuple, B<:NamedTuple, O<:Tuple{Vararg{FunctionalTables.ColumnOrdering,N} where N}, T<:FunctionalTable}

Implements by using an iterator.


Each rows from the underlying FunctionalTable is split into index and rest.

Iterator state is

  1. nothing when the rows of the underlying FunctionalTable have been exhausted,

  2. index, rest, itrstate for the next block, where index and rest are the first

(mismatching) row that has not been pushed to the buffers.

struct TrustLength

Wrapper type to indicate that the length should not be checked.


The perfect footgun. Only use when the lengths are known and verified by construction.

filter(f, ft; cfg)

Filter a FunctionalTable using a prediate that maps rows (as NamedTuples) to Bool. Preserves type (and thus ordering) of the table.

first(ft, n; cfg)

A FunctionalTable of the first n rows.

Useful for previews and data exploration. Preserves type (and thus ordering) of the table.

getindex(ft, keep)

With a tuple of symbols returns FunctionalTable with a subset of the columns.

With a single symbol, return that column (an iterable).

[drop = spec] will keep all but the given columns, where spec is a Tuple of Symbols.


julia ft[(:a, :b)] ft[:a] ft[drop = (:a, :b)]

map(f, st; cfg)

Map a table split with by using f.

Specifically, f(index, table) receives the split index (a NamedTuple) and a FunctionalTable.

It is supposed to return an iterable that returns rows (can be a FunctionalTable). These will be prepended with the corresponding index, and collected into a FunctionalTable with cfg.

When f returns just a single row (eg aggregation), wrap by Ref to create a single-element iterable.

merge(f, ft; cfg, replace)

Map ft using f by rows, then merge the two. See map(f, ::FunctionalTable).

cfg is passed to map, replace governs replacement of overlapping columns in merge.

merge(a, b; replace)

Merge two FunctionalTables.

When replace == true, columns in the first one are replaced by second one, otherwise an error is thrown if column names overlap.

The second table can be specified as a NamedTuple of columns.

append1(v, elt)

Append elt to v, allocating a new vector and copying the contents.

Type of new collection is calculated using promote_type.

cancontain(T, elt)

Test if a collection of element type T can contain a new element elt without any loss of precision.

collect_column(cfg, itr)

Collect results from itr into a sink (using config cfg), then finalize and return the column.


len, columns, ordering_rule =

collect_columns(cfg, itr, ordering_rule)
collect_columns(cfg, itr, ordering_rule, known_types)

Collect results from itr, which are supposed to be NamedTuples with the same names, into sinks (using config cfg), then finalize and return

  1. the length,

  2. the NamedTuple of the columns, and

  3. the ordering rule (which is always ::TrustOrdering, by construction).

Determine the names and types from the first named tuple, using known_types as the narrowest types for the given columns.

Special rules for empty iterators

When itr is empty, use a known_types will be used to create empty columns, and only the TryOrdering rule will be narrowed to these. Other rule with more column names may cause an error in the callee, which is intentional.

finalize_sink(?, sink)

Convert sink to a column.

sink may share structure with the result and is not supposed to be used for saving any more elements.

fuse(f, index, tables)

Prepend the index as repeated columns to f(index, tables...).

make_sink(cfg, ?)

Create and return a sink using configuration cfg that stores elements of type T. When T is unkown, use Base.Bottom.

mask_ordering(ordering, keys)
mask_ordering(ordering, keys, invert)

When invert == false, keep the initial part of ordering that has keys in keys. Not having a key in keys invalidates the tail ordering from that point. This is useful for selecting subsets of orderings.

When invert == true, having a key in keys invalidates the ordering. This is useful for orderings of merged and dropped columns.

mask_try_ordering(ordering_rule, keys)

For TryOrdering, return a masked ordering rule that is is defined on keys so that comparisons make sense, otherwise return the original ordering_rule (that will just error for undefined keys).


Convert the argument to a narrower type if possible without losing precision.

!!! NOTE This function is not type stable, use only when new container types are determined.

split_namedtuple(?, nt)

Splits a named tuple in two, based on the names in splitter.

Returns two NamedTuples; the first one is ordered as splitter, the second one with the remaining values as in the original argument.

julia> split_namedtuple(NamedTuple{(:a, :c)}, (c = 1, b = 2, a = 3, d = 4))
((a = 3, c = 1), (b = 2, d = 4))
start_sinks(cfg, row, known_types)

Start sinks using row, using the default known_types when available.!_or_reallocateMethod
store!_or_reallocate(?, sink, elt)

Either store elt in sink (in which case the returned value is ≡ sink), or allocate a new sink that can do this, copy the contents, save elt and return that (then the returned value is ≢ sink).!_or_reallocate_rowMethod
store!_or_reallocate_row(cfg, sinks, elts)

Broadcast store!_or_rellocate for a compatible (named) tuple of sinks and elts. Return the (potentially) new sinks.


Process ordering specifications for columns (an iterable or possibly a TableOrdering), return a TableOrdering. Check for uniqueness (but not validity) of keys.

Accepted syntax:

  • :key, for ordering a column in ascending order,

  • :key => reverse, for ordering a column in descending order.

All functions which accept ordering specifications should use this, but the function itself is not part of the API.