Create an object that can be used for resampling predictions and corresponding targets.

The set of predictions has to be provided as a vector consisting of

  • probabilities,
  • vectors of probabilities summing up to 1,
  • or distributions that can be sampled from.

Consistency resampling can be performed with rand and rand!, as described in the documentation of Random.


julia> predictions = rand(100);

julia> consistent = Consistent(predictions);

julia> rand(consistent) isa Tuple{Float64,Bool}

julia> first(rand(consistent)) in predictions

One can use Random.Sampler to create a sampler that is optimized for the generation of multiple samples, e.g., by building an alias table:

julia> using Random: Sampler, GLOBAL_RNG

julia> sampler = Sampler(GLOBAL_RNG, Consistent(rand(100)));

julia> rand(sampler) isa Tuple{Float64,Bool}

However, for small number of samples the setup cost can outweigh the improved sampling efficiency.


Multiple samples are returned as an array of tuples of predictions and corresponding targets. If you prefer a tuple of arrays of predictions and targets instead, you can use a package such as StructArrays and perform in-place sampling:

julia> using StructArrays, Random

julia> predictions = Vector{Float64}(undef, 100);

julia> targets = Vector{Bool}(undef, 100);

julia> rand!(StructVector((predictions, targets)), Consistent(rand(100)));


Bröcker, J. and Smith, L.A., 2007. Increasing the reliability of reliability diagrams. Weather and forecasting, 22(3), pp. 651-661.