CalibrationErrors.jl

Estimation of calibration errors.

Overview

This package implements different estimators of the expected calibration error (ECE), the squared kernel calibration error calibration error (SKCE), and the unnormalized calibration mean embedding (UCME) in the Julia language.

Example

Calibration errors can be estimated from a data set of predicted probabilities and and a set of targets by executing

calibrationerror(estimator, predictions, targets)

The predictions can be provided as a vector of n vectors of predicted probabilities of length m or as a matrix of size (m, n), in which each of the n columns corresponds to predicted probabilities of the targets 1,…,m. The corresponding targets have to be provided as a vector of length n, in which every element is from the set 1,…,m.

Alternatively, it is possible to specify a tuple of predictions and targets or a vector of tuples of predictions and targets.

This package implements the estimator ECE of the ECE, the estimators BiasedSKCE, UnbiasedSKCE, and BlockUnbiasedSKCE for the SKCE, and UCME for the UCME.

Related packages

CalibrationErrorsDistributions.jl extends calibration error estimation in this package to more general probabilistic predictive models that output arbitrary probability distributions.

CalibrationTests.jl implements statistical hypothesis tests of calibration.

References

If you use CalibrationsErrors.jl as part of your research, teaching, or other activities, please consider citing the following publications:

Widmann, D., Lindsten, F., & Zachariah, D. (2019). Calibration tests in multi-class classification: A unifying framework. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019) (pp. 12257–12267).

Widmann, D., Lindsten, F., & Zachariah, D. (2019). Calibration tests beyond classification. To be presented at ICLR 2021.