API Reference

A blocked array structure that stores thread-specific data in blocks. This facilitates a micro domain-decompisition for shared-memory applications. Each thread operates on it's own block of data. This provides better performance scaling than multi-threaded loops

Fields

• blocks::Vector{AA}:
• block_layout::NTuple{D,Int}: number of blocks along each dimension
• global_blockranges::Array{NTuple{D,UnitRange{Int}},D}: Indexing/ranges of each block from the global perspective
• nhalo::Int: Number of halo regions, e.g. 2 entries along each dimension
• loop_limits::Vector{Vector{Int}}: Looping limits for convienence e.g. [ilo,ihi,jlo,jhi]
• globaldims::NTuple{D,Int}: Dimensions of the array if it were a simple Array{T,D}, e.g. (20,20)

Construct a BlockHaloArray

Arguments

• dims::NTuple{N,Int}: Array dimensions
• nhalo::Integer: Number of halo entries (equal in all dimensions)

Keyword Arguments

• nblocks::Integer: Number of blocks to divide the array into; default is nthreads()
• T:: Array number type; default is Float64

Construct a BlockHaloArray from a normal Array

An MPI-aware BlockHaloArray. The only difference between this and the plain BlockHaloArray is the addition of send/receive buffers that help MPI communication.

Fields

• blocks::Vector{AA}:
• block_layout::NTuple{D,Int}: number of blocks along each dimension
• global_blockranges::Array{NTuple{D,UnitRange{Int}},D}: Indexing/ranges of each block from the global perspective
• nhalo::Int: Number of halo regions, e.g. 2 entries along each dimension
• loop_limits::Vector{Vector{Int}}: Looping limits for convienence e.g. [ilo,ihi,jlo,jhi]
• globaldims::NTuple{D,Int}: Dimensions of the array if it were a simple Array{T,D}, e.g. (20,20)
• _global_halo_send_buf::Vector{Array{T,D}}: Buffers used to send across MPI ranks
• _global_halo_recv_buf::Vector{Array{T,D}}: Buffers used to receive across MPI ranks

flatten(A::AbstractBlockHaloArray) -> Array

Return a flattened version of a BlockHaloArray. This is a copy, since a view of the current block structure isn't possible.

copy!(dst, src) -> dst

Copy from a BlockHaloArray into an AbstractArray. The global dimensions of the BlockHaloArray must be the same as the AbstractArray

copy!(dst, src) -> dst

Copy from an AbstractArray into a BlockHaloArray. The global dimensions of the BlockHaloArray must be the same as the AbstractArray

Overload the getindex, or [], method for a BlockHaloArray. This is a 'flat' iterator of sorts, since the actual data within the BlockHaloArray is a series of separate blocks. Iterating through the array in this manner will be slower due to the additional arithmetic need to find the global to local index conversion for each block.

Overload the setindex, or A[I] = ... , method for a BlockHaloArray. This is a 'flat' iterator of sorts, since the actual data within the BlockHaloArray is a series of separate blocks. Iterating through the array in this manner will be slower due to the additional arithmetic needed to find the global to local index conversion for each block.

Find the optimal block layout in 2D given total number of tiles n

Find the optimal block layout in 3D given total number of tiles n

For a given single dimension, find the block-local index

block_layout(nblocks, N)

Determine the block layout based on the number of total blocks nblocks and the dimensionality N of the domain

Arguments

• nblocks::Integer
• N::Integer

Get the scalar block index give a global index tuple

Return all common denominators of n

domain_donor_ranges(block, nhal, halodims::NTuple{3, Int}) -> Dict

Get the 3D ranges for each donor region in the doman that sends data to neighbor block halo regions

domain_donor_ranges(block, nhal, halodims::NTuple{2, Int}) -> Dict

Get the 2D ranges for each donor region in the doman that sends data to neighbor block halo regions

domain_donor_ranges(block, nhal, halodims::NTuple{1, Int}) -> Dict

Get the 1D ranges for each donor region in the doman that sends data to neighbor block halo regions

domainview(A::BlockHaloArray, blockid) -> SubArray

Get the SubArray view of the domain region of a block in the BlockHaloArray.

Get the domain donor region at a particular location, e.g. :ilo for block blockid

Generate the SubArray views of each domain region in A::BlockHaloArray that are called "donor" views. These are the regions copied from in the halo exchange copy/update.

Generate the SubArray views of each halo region in A::BlockHaloArray that "reciever" views. These are the regions updated in the halo exchange copy/update.

Get the cartesian block index within the block layout. For example, in a (2,3) block tiling scheme, based on a given global index global_i, and the dimension-specific cummulative block_sizes, find the 2D index. The dim argument is used to ignore dimensions that are not include in the halodims.

get_block_ranges(dims::NTuple{N, Int}, nblocks::Integer) -> Array{NTuple{ndims,UnitRange{Int}}}

Get the ranges of each block (in the global space)

Arguments

• dims::NTuple{N, Int}: Dimensions of the flat array to split into blocks
• nblocks::Integer: Number of total blocks

Get the block-local index based on the given global index. The block_i value is the dimension-specific block cartesian index. Dimensions not included in the halodims are a special case where the global index is the same as the local index.

Get the neighbor block id's for a 3D decomposition (with periodic edges)

Get the neighbor block id's for a 2D decomposition (with periodic edges)

Get the neighbor block id's for a 1D decomposition (with periodic edges)

Get the neighbor block id's for a 3D decomposition

Get the neighbor block id's for a 2D decomposition

Get the neighbor block id's for a 1D decomposition

globalindices(A::BlockHaloArray, block_index, local_indices) -> global_indices

Given a block index and local coordinates, return the global indices

Example

julia> globalindices(A, 2, (3, 4)) -> (8, 10)

3D halo exchange mapping, e.g. donor => reciever block ID

2D halo exchange mapping, e.g. donor => reciever block ID

1D halo exchange mapping, e.g. donor => reciever block ID

halo_reciever_ranges(block, nhalo, halodims::NTuple{3, Int}) -> Dict

Get the 3D ranges for each halo region that recieves data from neighbor blocks

halo_reciever_ranges(block, nhalo, halodims::NTuple{2, Int}) -> Dict

Get the 2D ranges for each halo region that recieves data from neighbor blocks

halo_reciever_ranges(block, nhalo, halodims::NTuple{1, Int}) -> Dict

Get the 1D ranges for each halo region that recieves data from neighbor blocks

Get the halo region at a particular location, e.g. :ilo for block blockid

Get the upper indices for an array A given a number of halo entries nhalo

Get the upper indices for an array A given a number of halo entries nhalo. This version properly accounts for non-halo dimensions

Get the lower indices for an array A given a number of halo entries nhalo

Get the lower indices for an array A given a number of halo entries nhalo. This version properly accounts for non-halo dimensions

Get the block-local indices give a global index tuple

Create a dictionary mapping the threadid to the NUMA node.

Determine if the block is on the boundary

Arguments

• A::AbstractBlockHaloArray: The array in question
• blockid::Integer: The id of the block. This is normally associated with the thread id
• boundary::Symbol: Which boundary are we checking for? Examples include :ilo, :jhi, etc...

repartition(A::AbstractBlockHaloArray, nblocks) -> BlockHaloArray

Repartition the BlockHaloArray into a different block layout

split_count(N::Integer, n::Integer)

Return a vector of n integers which are approximately equally sized and sum to N. This borrows from https://juliaparallel.org/MPI.jl/latest/examples/06-scatterv/

update_block_ranges_with_non_halo_dims(block_ranges, axes_sizes, halo_dims)

Update the ranges in each block to include the non-halo dimensions

updateblockhalo!(A::BlockHaloArray, block_id::Integer, include_periodic_bc=false)

Copy data from the neighbor block into the current block's halo region

Arguments

• A: BlockHaloArray
• block_id::Integer: Block index
• include_periodic_bc: Update the halo regions that are on periodic boundaries

updatehalo!(A, include_periodic_bc=false)

Synchronize the halo regions within each block. This spawns tasks so that each thread/block copies from it's neighbor block's domain into the current block's halo region.

Arguments

• A: BlockHaloArray
• include_periodic_bc: Update the halo regions that are on periodic boundaries