References

Index

ChunkSplitters

ChunkSplitters.chunks — Function
chunks(itr; n::Union{Nothing, Integer}, size::Union{Nothing, Integer} [, split::Symbol=:batch])

Returns an iterator that splits the indices of itr into n-many chunks (if n is given) or into chunks of a certain size (if size is given). The keyword arguments n and size are mutually exclusive. The returned iterator can be used to process chunks of itr one after another or in parallel (e.g. with @threads).

The optional argument split can be :batch (default) or :scatter and determines the distribution of the indices among the chunks. If split == :batch, chunk indices will be consecutive. If split == :scatter, the range is scattered over itr.

If you need a running chunk index you can combine chunks with enumerate. In particular, enumerate(chunks(...)) can be used in conjuction with @threads.

The itr is usually some iterable, indexable object. The interface requires it to have firstindex, lastindex, and length functions defined, as well as ChunkSplitters.is_chunkable(::typeof(itr)) = true.

Examples

julia> using ChunkSplitters

julia> x = rand(7);

julia> collect(chunks(x; n=3))
3-element Vector{UnitRange{Int64}}:
 1:3
 4:5
 6:7

julia> collect(enumerate(chunks(x; n=3)))
3-element Vector{Tuple{Int64, UnitRange{Int64}}}:
 (1, 1:3)
 (2, 4:5)
 (3, 6:7)

julia> collect(chunks(1:7; size=3))
3-element Vector{UnitRange{Int64}}:
 1:3
 4:6
 7:7

Note that chunks also works just fine for OffsetArrays:

julia> using ChunkSplitters, OffsetArrays

julia> x = OffsetArray(1:7, -1:5);

julia> collect(chunks(x; n=3))
3-element Vector{UnitRange{Int64}}:
 -1:1
 2:3
 4:5

julia> collect(chunks(x; n=3, split=:scatter))
3-element Vector{StepRange{Int64, Int64}}:
 -1:3:5
 0:3:3
 1:3:4
ChunkSplitters.getchunk — Method
getchunk(itr, i::Integer; n::Union{Nothing,Integer}, size::Union{Nothing,Integer}[, split::Symbol=:batch])

Returns the range of indices of itr that corresponds to the i-th chunk. How the chunks are formed depends on the keyword arguments. See chunks for more information.

Example

If we have an array of 7 elements, and the work on the elements is divided into 3 chunks, we have (using the default split = :batch option):

julia> using ChunkSplitters

julia> x = rand(7);

julia> getchunk(x, 1; n=3)
1:3

julia> getchunk(x, 2; n=3)
4:5

julia> getchunk(x, 3; n=3)
6:7

And using split = :scatter, we have:

julia> using ChunkSplitters

julia> x = rand(7);

julia> getchunk(x, 1; n=3, split=:scatter)
1:3:7

julia> getchunk(x, 2; n=3, split=:scatter)
2:3:5

julia> getchunk(x, 3; n=3, split=:scatter)
3:3:6

We can also choose the chunk size rather than the number of chunks:

julia> using ChunkSplitters

julia> x = rand(7);

julia> getchunk(x, 1; size=3)
1:3

julia> getchunk(x, 2; size=3)
4:6

julia> getchunk(x, 3; size=3)
7:7
ChunkSplitters.is_chunkable — Method
is_chunkable(::T) :: Bool

Determines if a of object of type T is capable of being chunked. Overload this function for your custom types if that type is linearly indexable and supports firstindex, lastindex, and length.

Interface requirements

Compat

Support for this minimal interface requires version 2.2.0.

For the chunks and getchunk functions to work, the input value must overload ChunkSplitters.is_chunkable(::YourType) = true, and support the functions Base.firstindex, Base.lastindex, and Base.length.

For example:

julia> using ChunkSplitters

julia> struct MinimalInterface end

julia> Base.firstindex(::MinimalInterface) = 1

julia> Base.lastindex(::MinimalInterface) = 7

julia> Base.length(::MinimalInterface) = 7

julia> ChunkSplitters.is_chunkable(::MinimalInterface) = true

julia> x = MinimalInterface()
MinimalInterface()

julia> collect(chunks(x; n=3))
3-element Vector{UnitRange{Int64}}:
 1:3
 4:5
 6:7

julia> collect(chunks(x; n=3, split=:scatter))
3-element Vector{StepRange{Int64, Int64}}:
 1:3:7
 2:3:5
 3:3:6