ClimaComms

ClimaComms.ClimaCommsModule
ClimaComms

Abstracts the communications interface for the various CliMA components in order to:

  • support different computational backends (CPUs, GPUs)
  • enable the use of different backends as transports (MPI, SharedArrays, etc.), and
  • transparently support single or double buffering for GPUs, depending on whether the transport has the ability to access GPU memory.

Devices

Missing docstring.

Missing docstring for ClimaComms.CPUSingleThreading. Check Documenter's build log for details.

Missing docstring.

Missing docstring for ClimaComms.CPUMultiThreading. Check Documenter's build log for details.

ClimaComms.deviceFunction
ClimaComms.device()

Automatically determine the appropriate device to use, returning one of

By default, it will check if a functional CUDA installation exists, using CUDA if possible.

Behavior can be overridden by setting the CLIMACOMMS_DEVICE environment variable to either CPU or CUDA.

ClimaComms.array_typeFunction
ClimaComms.array_type(::AbstractDevice)

The base array type used by the specified device (currently Array or CuArray).

ClimaComms.@threadedMacro
@threaded device for ... end

A threading macro that uses Julia native threading if the device is a CPUMultiThreaded type, otherwise return the original expression without Threads.@threads. This is done to avoid overhead from Threads.@threads, and the device is used (instead of checking Threads.nthreads() == 1) so that this is statically inferred.

References

  • https://discourse.julialang.org/t/threads-threads-with-one-thread-how-to-remove-the-overhead/58435
  • https://discourse.julialang.org/t/overhead-of-threads-threads/53964
ClimaComms.@timeMacro
@time device expr

Device-flexible @time.

Lowers to

@time expr

for CPU devices and

CUDA.@time expr

for CUDA devices.

ClimaComms.@elapsedMacro
@elapsed device expr

Device-flexible @elapsed.

Lowers to

@elapsed expr

for CPU devices and

CUDA.@elapsed expr

for CUDA devices.

ClimaComms.@syncMacro
@sync device expr

Device-flexible @sync.

Lowers to

@sync expr

for CPU devices and

CUDA.@sync expr

for CUDA devices.

Contexts

ClimaComms.contextFunction
ClimaComms.context(device=device())

Construct a default communication context.

By default, it will try to determine if it is running inside an MPI environment variables are set; if so it will return a MPICommsContext; otherwise it will return a SingletonCommsContext.

Behavior can be overridden by setting the CLIMACOMMS_CONTEXT environment variable to either MPI or SINGLETON.

ClimaComms.graph_contextFunction
graph_context(context::AbstractCommsContext, 
    sendarray, sendlengths, sendpids, 
    recvarray, recvlengths, recvpids)

Construct a communication context for exchanging neighbor data via a graph.

Arguments:

  • context: the communication context on which to construct the graph context.
  • sendarray: array containing data to send
  • sendlengths: list of lengths of data to send to each process ID
  • sendpids: list of processor IDs to send
  • recvarray: array to receive data into
  • recvlengths: list of lengths of data to receive from each process ID
  • recvpids: list of processor IDs to receive from

This should return an AbstractGraphContext object.

Context operations

ClimaComms.initFunction
(pid, nprocs) = init(ctx::AbstractCommsContext)

Perform any necessary initialization for the specified backend. Return a tuple of the processor ID and the number of participating processors.

ClimaComms.mypidFunction
mypid(ctx::AbstractCommsContext)

Return the processor ID.

ClimaComms.iamrootFunction
iamroot(ctx::AbstractCommsContext)

Return true if the calling processor is the root processor.

ClimaComms.nprocsFunction
nprocs(ctx::AbstractCommsContext)

Return the number of participating processors.

ClimaComms.abortFunction
abort(ctx::CC, status::Int) where {CC <: AbstractCommsContext}

Terminate the caller and all participating processors with the specified status.

Collective operations

ClimaComms.barrierFunction
barrier(ctx::CC) where {CC <: AbstractCommsContext}

Perform a global synchronization across all participating processors.

ClimaComms.reduceFunction
reduce(ctx::CC, val, op) where {CC <: AbstractCommsContext}

Perform a reduction across all participating processors, using op as the reduction operator and val as this rank's reduction value. Return the result to the first processor only.

ClimaComms.reduce!Function
reduce!(ctx::CC, sendbuf, recvbuf, op)
reduce!(ctx::CC, sendrecvbuf, op)

Performs elementwise reduction using the operator op on the buffer sendbuf, storing the result in the recvbuf of the process. If only one sendrecvbuf buffer is provided, then the operation is performed in-place.

ClimaComms.allreduceFunction
allreduce(ctx::CC, sendbuf, op)

Performs elementwise reduction using the operator op on the buffer sendbuf, allocating a new array for the result. sendbuf can also be a scalar, in which case recvbuf will be a value of the same type.

ClimaComms.allreduce!Function
allreduce!(ctx::CC, sendbuf, recvbuf, op)
allreduce!(ctx::CC, sendrecvbuf, op)

Performs elementwise reduction using the operator op on the buffer sendbuf, storing the result in the recvbuf of all processes in the group. Allreduce! is equivalent to a Reduce! operation followed by a Bcast!, but can lead to better performance. If only one sendrecvbuf buffer is provided, then the operation is performed in-place.

ClimaComms.bcastFunction
bcast(ctx::AbstractCommsContext, object)

Broadcast object from the root process to all other processes. The value of object on non-root processes is ignored.

Graph exchange

ClimaComms.startFunction
start(ctx::AbstractGraphContext)

Initiate graph data exchange.

ClimaComms.progressFunction
progress(ctx::AbstractGraphContext)

Drive communication. Call after start to ensure that communication proceeds asynchronously.

ClimaComms.finishFunction
finish(ctx::AbstractGraphContext)

Complete the communications step begun by start(). After this returns, data received from all neighbors will be available in the stage areas of each neighbor's receive buffer.