StochasticRounding.jl

Stochastic rounding for floating-point arithmetic.

This package exports Float32sr,Float16sr, and BFloat16sr, three number formats that behave like their deterministic counterparts but with stochastic rounding that is proportional to the distance of the next representable numbers and therefore exact in expectation (see also example below in Usage. The only known hardware implementation available is Graphcore's IPU with stochastic rounding, but other vendors are likely working on stochastic rounding for their next-generation GPUs (and maybe CPUs too).

The software emulation of stochastic rounding in StochasticRounding.jl makes the number format slower, but e.g. Float32+stochastic rounding is only about 2x slower than Float64. Xoroshiro128Plus, a random number generator from the Xorshift family, is used through the RandomNumbers.jl package, due to its speed and statistical properties.

You are welcome to raise issues, ask questions or suggest any changes or new features.

BFloat16sr is based on BFloat16s.jl
Float16sr is slow in Julia <1.6, but fast in Julia >=1.6 due to LLVM's half support.

Usage

Float32sr, Float16sr and BFloat16sr are supposed to be drop-in replacements for their deterministically rounded counterparts. You can create data of those types as expected (which is bitwise identical to the deterministic formats respectively) and the type will trigger stochastic rounding on every arithmetic operation.

julia> a = BFloat16sr(1.0)
BFloat16sr(1.0)
julia> a/3
BFloat16sr(0.33398438)
julia> a/3
BFloat16sr(0.33203125)

As 1/3 is not exactly representable the rounding will be at 66.6% chance towards 0.33398438 and at 33.3% towards 0.33203125 such that in expectation the result is 0.33333... and therefore exact. You can use BFloat16_chance_roundup(x::Float32) to get the chance that x will be round up.

Solving a linear equation system with LU decomposition and stochastic rounding:

A = Float32sr.(randn(3,3))
b = Float32sr.(randn(3))

Now execute the \ several times and the results will differ slightly due to stochastic rounding

julia> A\b
3-element Vector{Float32sr}:
  3.3321106f0
  2.0391452f0
 -0.59199476f0

julia> A\b
3-element Vector{Float32sr}:
  3.3321111f0
  2.0391457f0
 -0.5919949f0

The random number generator is randomly seeded on every import or using such that running the same calculations twice, will not yield bit-reproducible results. However, you can seed the random number generator at any time with any integer larger than zero as follows

julia> StochasticRounding.seed(2156712)

Theory

Round-to-nearest (tie to even) is the standard rounding mode for IEEE floats. Stochastic rounding is explained in the following schematic

The exact result x of an arithmetic operation (located at one fifth between x₂ and x₃ in this example) is always rounded down to x₂ for round-to-nearest. For stochastic rounding, only at 80% chance x is round down. At 20% chance it is round up to x₃, proportional to the distance of x between x₂ and x₃.

Installation

StochasticRounding.jl is registered in the Julia registry. Hence, simply do

julia>] add StochasticRounding

where ] opens the package manager.

Performance

StochasticRounding.jl is among the fastest software implementation of stochastic rounding for floating-point arithmetic. Define a few random 1000000-element arrays

julia> using StochasticRounding, BenchmarkTools, BFloat16s
julia> A = rand(Float64,1000000);
julia> B = rand(Float64,1000000);   # A, B shouldn't be identical as a+a=2a is not round

And similarly for the other number types. Then with Julia 1.6 on an Intel(R) Core(R) i5 (Ice Lake) @ 1.1GHz timings via @btime +($A,$B) are

Stochastic rounding imposes an about x5 performance decrease for Float32 and BFloat16, but only x2 for Float16. For more complicated benchmarks the performance decrease is usually within x10. About 50% of the time is spend on the random number generation with Xoroshiro128+.

Citation

If you use this package please cite us

Paxton EA, M Chantry, M Klöwer, L Saffin, TN Palmer, 2022. Climate Modelling in Low-Precision: Effects of both Deterministic & Stochastic Rounding, Journal of Climate, 10.1175/JCLI-D-21-0343.1