LazyStack.jl

Travis CI Github CI

This package exports one function, stack, for turning a list of arrays into one AbstractArray. Given several arrays with the same eltype, or an array of such arrays, it returns a lazy Stacked{T,N} view of these:

stack([zeros(2,2), ones(2,2)])  # isa Stacked{Float64, 3, <:Vector{<:Matrix}}
stack([1,2,3], 4:6)             # isa Stacked{Int, 2, <:Tuple{<:Vector, <:UnitRange}}

Given a generator, it instead iterates through the elements and writes into a new array. Given a function and then some arrays, it behaves like map(f, A, B) but immediately writes into a new array:

stack([i,2i] for i in 1:5)            # isa Matrix{Int}     # size(ans) == (2, 5)
stack(*, eachcol(ones(2,4)), 1:4)     # == Matrix(stack(map(*, eachcol(...), 1:4)))

The same stack_iter method is also used for any list of arrays of heterogeneous element type, and for arrays of tuples. Notice that like map(identity, Any[1, 1.0, 5im]), this promotes using promote_typejoin, to Number here, rather than to Complex{Float64}:

stack([1,2], [3.0, 4.0], [5im, 6im])  # isa Matrix{Number}  # size(ans) == (2, 3)
stack([(i,2.0,3//j) for i=1:4, j=1:5])# isa Array{Real, 3}  # size(ans) == (3, 4, 5)

The slices must all have the same size, but they (and the container) can have any number of dimensions. stack always places the slice dimensions first. There are no options.

Ragged stack

There is also a version which does not demand that slices have equal size (or equal ndims), which always returns a new Array. You can control the position of slices using OffsetArrays:

rstack([1:n for n in 1:10])           # upper triangular Matrix{Int}
rstack(OffsetArray(fill(n,4), rand(-2:2)) for n in 1:10; fill=NaN)

Other packages

This one plays well with OffsetArrays.jl, NamedDims.jl, and Zygote.jl.

Besides which, there are several other ways to achieve similar things:

  • For an array of arrays, you can also use JuliennedArrays.Align. This requires (or enables) you to specify which dimensions of the output belong to the sub-arrays, instead of writing PermutedDimsArray(stack(...), ...).
  • There is also RecursiveArrayTools.VectorOfArray which as its name hints only allows a one-dimensional container. Linear indexing retreives a slice, not an element, which is sometimes surprising.
  • For a tuple of arrays, LazyArrays.Hcat is at present faster to index than stack, but doesn't allow arbitrary dimensions.
  • For a generator of arrays, the built-in reduce(hcat,...) may work, but it slow compared to stack: see test/speed.jl for some examples.

And a few more:

  • When writing this I missed SplitApplyCombine.combinedimsview, which is very similar to stack, but doesn't handle tuples.
  • Newer than this package is StackViews.jl handles both, with StackView(A,B,dims=4) == StackView([A,B],4) creating a 4th dimension; the container is always one-dimensional.
  • Flux.stack similarly takes a dimension, but eagerly creates an Array.

The lazy inverse:

  • The package ArraysOfArrays.jl solves the opposite problem, of accessing one large array as if it were many slices.

  • As does JuliennedArrays.Slices.

  • As does PackedVectorsOfVectors, although only 1+1 dimensions. Also has an eager pack method which turns a vector of vectors into view of a single larger matrix.

  • Base.eachslice also views one large array as many slices. This is a generator, but JuliaLang#32310 should upgrade it to a multi-dimensional container indexable container.

Eager:

  • After writing this I learned of JuliaLang#31644 which extends reduce(hcat,...) to work on generators.

  • Later, JuliaLang#31644 proposes to add the eager stack_iter method of this package to Base.