CalibrationErrors.jl
Estimation of calibration errors.
There are also Python and R interfaces for this package
Overview
This package implements different estimators of the expected calibration error (ECE), the squared kernel calibration error (SKCE), and the unnormalized calibration mean embedding (UCME) in the Julia language.
This package supports calibration error estimation of classification models that output vectors of class probabilities. In addition, SKCE and UCME can be estimated for more general probabilistic predictive models that output probability distributions defined in Distributions.jl such as normal and Laplace distributions.
Example
Calibration errors can be estimated from a data set of predicted probability distributions and a set of corresponding observed targets by executing
estimator(predictions, targets)
The sets of predictions and targets have to be provided as vectors.
This package implements the estimator ECE
of the ECE, the estimator SKCE
for the SKCE
(unbiased and biased variants with different sample complexity), and UCME
for the UCME.
Related packages
CalibrationTests.jl implements statistical hypothesis tests of calibration.
pycalibration is a Python interface for CalibrationErrors.jl and CalibrationTests.jl.
rcalibration is an R interface for CalibrationErrors.jl and CalibrationTests.jl.
Talk at JuliaCon 2021
The slides of the talk are available as Pluto notebook.
References
If you use CalibrationsErrors.jl as part of your research, teaching, or other activities, please consider citing the following publications:
Widmann, D., Lindsten, F., & Zachariah, D. (2019). Calibration tests in multi-class classification: A unifying framework. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019) (pp. 12257–12267).
Widmann, D., Lindsten, F., & Zachariah, D. (2021). Calibration tests beyond classification. International Conference on Learning Representations (ICLR 2021).