ExpandNestedData.jl

ExpandNestedData.jl is a small package that can consume nested data structures like dictionaries of dictionaries or structs of structs and produce a normalized, Tables.jl-compliant NamedTuple. It can be used with JSON3.jl, XMLDict.jl, and other packages that parse file formats which are structured as denormalized data. It operates similarly to Pandas.json_normalize, but it is much more flexible.

Getting Started

Install

using Pkg
Pkg.add("ExpandNestedData")

Basic Usage

ExpandNestedData provides a single function expand to flatten out nested data.

using JSON3
using DataFrames

message = JSON3.read("""
    {
        "a" : [
            {"b" : 1, "c" : 2},
            {"b" : 2},
            {"b" : [3, 4], "c" : 1},
            {"b" : []}
        ],
        "d" : 4
    }
    """
)

expand(message) |> DataFrame

Configuring Options

While expand can produce a Table out-of-the-box, it is often useful to configure some options in how it handles the normalization process. ExpandNestedData.jl offers two ways to set these configurations. You can set them at the table-level with expand's keyword arguments or exercise finer control with per-column configurations.

Keyword Arguments

name_map = Dict([:a, :b] => :Column_B)
expand(message; default_value="no value", pool_arrays=true, column_names=name_map) |> DataFrame

Using ColumnDefinitions

Instead of setting the configurations for the whole dataset, you can use a Vector{ColumnDefinition} to control how each column is handled. Using ColumnDefinitions has the added benefit of allowing you to ignore certain fields from the input. ColumnDefinition takes a Vector or Tuple of keys that act as the path to the values for the column. It also supports most of the keyword arguments as the regular expand API with the following exceptions:

• column_names is column_name and accepts a single Symbol
• No support for lazy_columns
• column_style does not apply

column_defs = [
    ColumnDefinition([:d]; column_name = :ColumnD),
    ColumnDefinition([:a, :b]),
    ColumnDefinition([:e, :f]; column_name = :MissingColumn, default_value="Missing branch")
]

expand(message, column_defs) |> DataFrame

ColumnStyles

In the examples above, we've used flat_columns style. However, we can also maintain the nesting hierarchy of the source data.

using TypedTables

tbl = expand(message; column_style = nested_columns)

Now, our table has its columns nested, so we can access a specific column using dot syntax.

tbl.a.b

Furthermore, rows(tbl) returns a nested NamedTuple for each row

tbl |> rows |> first

API