CalibrationErrors.jl

Bin(predictions, targets)

Create bin of predictions and corresponding targets.

Bin(prediction, target)

Create bin of a signle prediction and corresponding target.

ECE(binning[, distance = TotalVariation()])

Create an estimator of the expected calibration error (ECE) with the given binning algorithm and distance function.

adddata!(bin::Bin, prediction, target)

Update running statistics of the bin by integrating one additional pair of predictions and target.

calibrationerror(estimator::CalibrationErrorEstimator, data...)

Estimate the calibration error of a model from the data set of predictions and corresponding targets using the estimator.

The data can be a tuple of predictions and targets or an array of tuples of predictions and targets.

unsafe_skce_eval(k, p, y, p̃, ỹ)

Evaluate

\[k((p, y), (p̃, ỹ)) - E_{z ∼ p}[k((p, z), (p̃, ỹ))] - E_{z̃ ∼ p̃}[k((p, y), (p̃, z̃))] + E_{z ∼ p, z̃ ∼ p̃}[k((p, z), (p̃, z̃))]\]

for kernel k and predictions p and p̃ with corresponding targets y and ỹ.

This method assumes that p, p̃, y, and ỹ are valid and specified correctly, and does not perform any checks.

evaluate(distance, bin::Bin)

Evaluate the distance between the average prediction and the distribution of targets in the bin.