File(parent, name, value=NoValue())

Fields:

• parent::Union{FileTree, Nothing} – The parent node. nothing if it's the root node.
• name::String – Name of the root.
• value::Any – the value at the node, if no value is present, a NoValue() sentinal value.

FileTree(parent, name, children, value)

Fields:

• parent::Union{FileTree, Nothing} – The parent node. nothing if it's the root node.

• name::String – Name of the root.

• children::Vector – children

• value::Any – the value at the node, if no value is present, a NoValue() sentinal value.

  FileTree(tree::FileTree; parent, name, children, value)

Copy over all fields from tree, but use any fields provided as keyword arguments.

FileTree(dirname::String)

Construct a FileTree to reflect directory from disk in the current working directory.

A wrapper around function f which causes overwriter functions (combine option) to get the nodes themselves rather than the value stored in them.

Note that if the nodes have lazy values, the values will be Thunks.

children(node::Union{FileTree, File})

Get the immediate children of a FileTree node. If node is File then returns ().

cp(t::FileTree,
   from_path::Regex,
   to_path::SubstitutionString; combine, associative=false)

copy nodes in the file tree whose path matches the from_tree regular expression pattern by renaming it to to_path pattern. Any sub-pattern in from_path which is surrounded by paranthesis will be read as a matched substring which can be accessed in the to_path substitution pattern using \1, \2 etc. positional matches.

If a file overwrites an existing node after copy, combine will be called to combine them together. By default combine will error.

Use associative=true if combine is associative to improve parallelism.

Example:

julia> t = maketree("dir" => [string(j) => [string(i)=>["data.csv"] for i = 1:2] for j=1:2])
dir/
├─ 1/
│  ├─ 1/
│  │  └─ data.csv
│  └─ 2/
│     └─ data.csv
└─ 2/
   ├─ 1/
   │  └─ data.csv
   └─ 2/
      └─ data.csv

julia> cp(t, r"^([^/]*)/([^/]*)/data.csv$", s"
/.csv")
dir/
├─ 1/
│  ├─ 1/
│  │  └─ data.csv
│  ├─ 1.csv
│  ├─ 2/
│  │  └─ data.csv
│  └─ 2.csv
└─ 2/
   ├─ 1/
   │  └─ data.csv
   ├─ 1.csv
   ├─ 2/
   │  └─ data.csv
   └─ 2.csv

mkpath(t::FileTree, path::AbstractString; value)

Create a directory node at path in the tree. Does not contain any value by default.

mv(t::FileTree,
   from_path::Regex,
   to_path::SubstitutionString; combine, associative=false)

move nodes in the file tree whose path matches the from_tree regular expression pattern by renaming it to to_path pattern. Any sub-pattern in from_path which is surrounded by paranthesis will be read as a matched substring which can be accessed in the to_path substitution pattern using \1, \2 etc. positional matches.

If a file overwrites an existing node after copy, combine will be called to combine them together. By default combine will error.

Use associative=true if combine is associative to improve parallelism.

Example:

julia> t = maketree("dir" => [string(j) =>
                                [string(i)=>["data.csv"]
                                    for i = 1:2] for j=1:2])
dir/
├─ 1/
│  ├─ 1/
│  │  └─ data.csv
│  └─ 2/
│     └─ data.csv
└─ 2/
   ├─ 1/
   │  └─ data.csv
   └─ 2/
      └─ data.csv

julia> mv(t, r"^([^/]*)/([^/]*)/data.csv$", s"\1/\2.csv")
dir/
├─ 1/
│  ├─ 1.csv
│  └─ 2.csv
└─ 2/
   ├─ 1.csv
   └─ 2.csv

rm(t::FileTree, pattern::Union{Glob, String, AbstractPath, Regex})

remove nodes which match pattern from the file tree.

touch(t::FileTree, path::AbstractString; value)

Create an file node at path in the tree. Does not contain any value by default.

diff(t1::FileTree, t2::FileTree)

For each node in t2 remove a node in t1 at the same path if it exists. Returns the difference tree.

filter(f, tree::FileTree; walk=FileTrees.postwalk, dirs=true)

Return a copy of tree, removing nodes for which f is false.

The function f is passed all nodes (Files and FileTrees) if dirs=true and leaf nodes (Files) if dirs=false.

walk can be either FileTrees.postwalk or FileTrees.postwalk.

get(node)

Get the value stored in the node. NoValue() is returned if there is no value stored.

map(f, tree::FileTree; walk=FileTrees.postwalk, dirs=true)

apply f to every node in the tree. To only visit File nodes, pass dirs=false.

walk can be either FileTrees.postwalk or FileTrees.postwalk.

merge(t1::FileTree, t2::FileTree; combine)

Merge two FileTrees. If files at the same path contain values, the combine callback will be called with their values to result in a new value.

If one of the dirs does not have a value, its corresponding argument will be NoValue() If any of the values is lazy, the output value is lazy as well.

parent(node::Union{FileTree, File})

Get the parent node. Returns nothing if there are no parents.

values(tree::FileTree; dirs=true)

Get a vector of all non-null values from nodes in the tree.

dirs=false will exclude any value stored in FileTree sub nodes.

compute(tree::FileTree; cache=true)

Compute any lazy values (Thunks) in tree and return a new tree where the values refer to the computed values (maybe on remote processes). The tree still behaves as a Lazy tree. exec on it will fetch the values from remote processes.

Path(file::Union{File, FileTree)

Returns an AbstractPath object which is the Path of the file from the root node leading up to this file.

Associative reduce

clip(t, n; combine)

Remove n top-level directories. combine will be called to merge any nodes with equal names found at any level being clipped.

dirs(tree::FileTree, dirs=true)

Get a vector of all directories in the tree.

exec(ctx, x)

Same as exec(x) with a ctx being passed to Dagger when computing any Thunks.

exec(x)

If x is a FileTree, computes any uncomputed Thunks stored as values in it. Returns a new tree with the computed values. If x is a File, computes the value if it is a Thunk. Returns a new File with the computed value. If x is a Thunk (such as the result of a reducevalues), then exec will compute the result. If x is anything else, exec just returns the same value.

files(tree::FileTree)

Get a vector of all files in the tree.

get(node)

Get the value stored in the node. NoValue() is returned if there is no value stored.

load(f, t::FileTree; dirs=false)

Walk the tree and optionally load data for nodes in it.

f(file) is the loader function which takes File as input. Call path(file) to get the String path to read the file.

If dirs = true then f can either get a File or FileTree. nodes within FileTree will have already been loaded.

If NoValue() is returned by f, no value is attached to the node. hasvalue(x) tells you if x already has a value or not.

mapsubtrees(f, t::FileTree, pattern::Union{GlobMatch, Regex})

For every node that matches the pattern provided, apply the function f.

If f returns either a File or FileTree, this new node will replace the matched node.

If f returns nothing, the matched node will be deleted

If f returns any other value, the value will be used as the value of the node and the node itself will be emptied of children.

This will allow use of mapsubtrees for complex use cases.

Suppose you would like to combine the values of a subdirectory with the function hcat and in turn those values using vcat, you can use mapsubtrees to accomplish this:

Here is a demo:

t = maketree("dir"=>([string(j)=>[(name=string(i), value=(i,j)]
                        for i=1:2] for j=1:3]

t1 = mapsubtrees("*") do subtree
    reducevalues(vcat, subtree)
end

reducevalues(hcat, t1)

apply f to any node that has a value. f gets the node itself and must return a node.

mapvalues(f, x::FileTree)

(See load to load values into nodes of a tree.)

Apply f to the value of all nodes in x which have a value. Returns a new tree where every value is replaced with the result of applying f.

f may return NoValue() to cause no value to be associated with a node.

name(node::Union{FileTree, File})

Get the file or directory name.

nodes(tree::FileTree, dirs=true)

Get a vector of all nodes in the tree.

dirs=false will return only File nodes.

reducevalues(f, t::FileTree; associative=true, init=nothing)

Use f to combine values in the tree.

• associative=true assumes f can be applied in an associative way
• init keyword argument will be returned if the file tree is empty. if init is not provided, reduce over an empty tree will cause an error to be thrown

rename(node::Union{FileTree, File}, newname)

Return a copy of node with name set to newname.

save(f, x::Node)

Save a FileTree to disk. Creates the directory structure and calls f with File for every file in the tree which has a value associated with it.

(see load and mapvalues for associating values with files.)

setparent(node::Union{FileTree, File}, parent)

Return a copy of node with parent set as the parent.

setvalue(node::Union{FileTree, File}, val)

Return a copy of node with val set as the value.

values(tree::FileTree; dirs=true)

Get a vector of all non-null values from nodes in the tree.

dirs=false will exclude any value stored in FileTree sub nodes.