Private API

Abstract Data Transformer


The supertype for methods producing or consuming data.

                 ╵               ▼
Storage ◀────▶ Data          Information
                 ▲               ╷

There are three subtypes:

  • DataStorage
  • DataLoader
  • DataWrite

Each subtype takes a Symbol type parameter designating the driver which should be used to perform the data operation. In addition, each subtype has the following fields:

  • dataset::DataSet, the data set the method operates on
  • type::Vector{<:QualifiedType}, the Julia types the method supports
  • priority::Int, the priority with which this method should be used, compared to alternatives. Lower values have higher priority.
  • parameters::SmallDict{String, Any}, any parameters applied to the method.

Advice Amalgamation


A collection of Advices sourced from available Plugins.

Like individual Advices, a AdviceAmalgamation can be called as a function. However, it also supports the following convenience syntax:

(::AdviceAmalgamation)(f::Function, args...; kargs...) # -> result


AdviceAmalgamation(adviseall::Function, advisors::Vector{Advice},
                   plugins_wanted::Vector{String}, plugins_used::Vector{String})

Qualified Types


A representation of a Julia type that does not need the type to be defined in the Julia session, and can be stored as a string. This is done by storing the type name and the module it belongs to as Symbols.


While QualifiedType is currently quite capable, it is not currently able to express the full gamut of Julia types. In future this will be improved, but it will likely always be restricted to a certain subset.


While the subtype operator cannot work on QualifiedTypes (<: is a built-in), when the Julia types are defined the subset operator can be used instead. This works by simply converting the QualifiedTypes to the corresponding Type and then applying the subtype operator.

julia> QualifiedTypes(:Base, :Vector) ⊆ QualifiedTypes(:Core, :Array)

julia> Matrix ⊆ QualifiedTypes(:Core, :Array)

julia> QualifiedTypes(:Base, :Vector) ⊆ AbstractVector

julia> QualifiedTypes(:Base, :Foobar) ⊆ AbstractVector


QualifiedType(parentmodule::Symbol, typename::Symbol)


A QualifiedType can be expressed as a string as "$parentmodule.$typename". This can be easily parsed as a QualifiedType, e.g. parse(QualifiedType, "Core.IO").

Global variables


The set of packages loaded by each module via @addpkg, for import with @import.

More specifically, when a module M invokes @addpkg pkg id then EXTRA_PACKAGES[M][pkg] = id is set, and then this information is used with @import to obtain the package from the root module.


The data specification TOML format constructs a DataCollection, which itself contains DataSets, comprised of metadata and AbstractDataTransformers.

├─ DataSet
│  ├─ AbstractDataTransformer
│  └─ AbstractDataTransformer
├─ DataSet

Within each scope, there are certain reserved attributes. They are listed in this Dict under the following keys:

  • :collection for DataCollection
  • :dataset for DataSet
  • :transformer for AbstractDataTransformer