ConformalPrediction.available_models
— ConstantA container listing all available methods for conformal prediction.
ConformalPrediction.tested_atomic_models
— ConstantA container listing all atomic MLJ models that have been tested for use with this package.
ConformalPrediction.AdaptiveInductiveClassifier
— TypeThe AdaptiveInductiveClassifier
is an improvement to the SimpleInductiveClassifier
and the NaiveClassifier
. Contrary to the NaiveClassifier
it computes nonconformity scores using a designated calibration dataset like the SimpleInductiveClassifier
. Contrary to the SimpleInductiveClassifier
it utilizes the softmax output of all classes.
ConformalPrediction.CVMinMaxRegressor
— TypeConstructor for CVMinMaxRegressor
.
ConformalPrediction.CVPlusRegressor
— TypeConstructor for CVPlusRegressor
.
ConformalPrediction.ConformalInterval
— TypeAn abstract base type for conformal models that produce interval-valued predictions. This includes most conformal regression models.
ConformalPrediction.ConformalProbabilistic
— TypeAn abstract base type for conformal models that produce probabilistic predictions. This includes some conformal classifier like Venn-ABERS.
ConformalPrediction.ConformalProbabilisticSet
— TypeAn abstract base type for conformal models that produce set-valued probabilistic predictions. This includes most conformal classification models.
ConformalPrediction.ConformalQuantileRegressor
— TypeConstructor for ConformalQuantileRegressor
.
ConformalPrediction.JackknifeMinMaxRegressor
— TypeConstructor for JackknifeMinMaxRegressor
.
ConformalPrediction.JackknifePlusAbMinMaxRegressor
— TypeConstructor for JackknifePlusAbMinMaxRegressor
.
ConformalPrediction.JackknifePlusAbRegressor
— TypeConstructor for JackknifePlusAbPlusRegressor
.
ConformalPrediction.JackknifePlusRegressor
— TypeConstructor for JackknifePlusRegressor
.
ConformalPrediction.JackknifeRegressor
— TypeConstructor for JackknifeRegressor
.
ConformalPrediction.NaiveClassifier
— TypeThe NaiveClassifier
is the simplest approach to Inductive Conformal Classification. Contrary to the NaiveClassifier
it computes nonconformity scores using a designated training dataset.
ConformalPrediction.NaiveRegressor
— TypeThe NaiveRegressor
for conformal prediction is the simplest approach to conformal regression.
ConformalPrediction.QuantileModel
— TypeUnion type for quantile models.
ConformalPrediction.SimpleInductiveClassifier
— TypeThe SimpleInductiveClassifier
is the simplest approach to Inductive Conformal Classification. Contrary to the NaiveClassifier
it computes nonconformity scores using a designated calibration dataset.
ConformalPrediction.SimpleInductiveRegressor
— TypeThe SimpleInductiveRegressor
is the simplest approach to Inductive Conformal Regression. Contrary to the NaiveRegressor
it computes nonconformity scores using a designated calibration dataset.
ConformalPrediction.TimeSeriesRegressorEnsembleBatch
— TypeConstructor for TimeSeriesRegressorEnsembleBatch
.
ConformalPrediction._aggregate
— Method_aggregate(y, aggregate::Union{Symbol,String})
Helper function that performs aggregation across vector of predictions.
ConformalPrediction.absolute_error
— Methodabsolute_error(y,ŷ)
Computes abs(y - ŷ)
where ŷ
is the predicted value.
ConformalPrediction.blockbootstrap
— Methodblockbootstrap(time_series_data, block_szie)
Generate a sampling method, that block bootstraps the given data
ConformalPrediction.conformal_model
— Methodconformal_model(model::Supervised; method::Union{Nothing, Symbol}=nothing, kwargs...)
A simple wrapper function that turns a model::Supervised
into a conformal model. It accepts an optional key argument that can be used to specify the desired method
for conformal prediction as well as additinal kwargs...
specific to the method
.
ConformalPrediction.emp_coverage
— Methodemp_coverage(ŷ, y)
Computes the empirical coverage for conformal predictions ŷ
.
ConformalPrediction.ineff
— Functionineff(ŷ)
Computes the inefficiency (average set size) for conformal predictions ŷ
.
ConformalPrediction.is_classification
— Methodis_classification(ŷ)
Helper function that checks if conformal prediction ŷ
comes from a conformal classification model.
ConformalPrediction.is_covered
— Methodis_covered(ŷ, y)
Helper function to check if y
is contained in conformal region. Based on whether conformal predictions ŷ
are set- or interval-valued, different checks are executed.
ConformalPrediction.is_covered_interval
— Methodis_covered_interval(ŷ, y)
Helper function to check if y
is contained in conformal interval.
ConformalPrediction.is_covered_set
— Methodis_covered_set(ŷ, y)
Helper function to check if y
is contained in conformal set.
ConformalPrediction.is_regression
— Methodis_regression(ŷ)
Helper function that checks if conformal prediction ŷ
comes from a conformal regression model.
ConformalPrediction.minus_softmax
— Methodminus_softmax(y,ŷ)
Computes 1.0 - ŷ
where ŷ
is the softmax output for a given class.
ConformalPrediction.partial_fit
— Functionpartial_fit(conf_model::TimeSeriesRegressorEnsembleBatch, fitresult, X, y, shift_size)
For the TimeSeriesRegressorEnsembleBatch
Non-conformity scores are updated by the most recent data (X,y). shift_size determines how many points in Non-conformity scores will be discarded.
ConformalPrediction.qminus
— Functionqminus(v::AbstractArray, coverage::AbstractFloat=0.9)
Implements the $\hat{q}_{n,\alpha}^{-}$ finite-sample corrected quantile function as defined in Barber et al. (2020): https://arxiv.org/pdf/1905.02928.pdf.
ConformalPrediction.qplus
— Functionqplus(v::AbstractArray, coverage::AbstractFloat=0.9)
Implements the $\hat{q}_{n,\alpha}^{+}$ finite-sample corrected quantile function as defined in Barber et al. (2020): https://arxiv.org/pdf/1905.02928.pdf.
ConformalPrediction.reformat_interval
— Methodreformat_interval(ŷ)
Reformats conformal interval predictions.
ConformalPrediction.reformat_mlj_prediction
— Methodreformat_mlj_prediction(ŷ)
A helper function that extracts only the output (predicted values) for whatever is returned from MMI.predict(model, fitresult, Xnew)
. This is currently used to avoid issues when calling MMI.predict(model, fitresult, Xnew)
in pipelines.
ConformalPrediction.score
— Functionscore(conf_model::ConformalProbabilisticSet, fitresult, X, y=nothing)
Generic score method for the ConformalProbabilisticSet
. It computes nonconformity scores using the heuristic function h
and the softmax probabilities of the true class. Method is dispatched for different Conformal Probabilistic Sets and atomic models.
ConformalPrediction.score
— Functionscore(conf_model::SimpleInductiveClassifier, ::Type{<:Supervised}, fitresult, X, y::Union{Nothing,AbstractArray}=nothing)
Score method for the SimpleInductiveClassifier
dispatched for any <:Supervised
model.
ConformalPrediction.score
— Functionscore(conf_model::AdaptiveInductiveClassifier, ::Type{<:Supervised}, fitresult, X, y::Union{Nothing,AbstractArray}=nothing)
Score method for the AdaptiveInductiveClassifier
dispatched for any <:Supervised
model.
ConformalPrediction.set_size
— Methodset_size(ŷ)
Helper function that computes the set size for conformal predictions.
ConformalPrediction.size_stratified_coverage
— Methodsize_stratified_coverage(ŷ, y)
Computes the size-stratified coverage for conformal predictions ŷ
.
ConformalPrediction.split_data
— Methodsplit_data(conf_model::ConformalProbabilisticSet, indices::Base.OneTo{Int})
Splits the data into a proper training and calibration set.
MLJModelInterface.fit
— MethodMMI.fit(conf_model::AdaptiveInductiveClassifier, verbosity, X, y)
For the AdaptiveInductiveClassifier
nonconformity scores are computed by cumulatively summing the ranked scores of each label in descending order until reaching the true label $Y_i$:
$S_i^{\text{CAL}} = s(X_i,Y_i) = \sum_{j=1}^k \hat\mu(X_i)_{\pi_j} \ \text{where } \ Y_i=\pi_k, i \in \mathcal{D}_{\text{calibration}}$
MLJModelInterface.fit
— MethodMMI.fit(conf_model::CVMinMaxRegressor, verbosity, X, y)
For the CVMinMaxRegressor
nonconformity scores are computed in the same way as for the CVPlusRegressor
. Specifically, we have,
$S_i^{\text{CV}} = s(X_i, Y_i) = h(\hat\mu_{-\mathcal{D}_{k(i)}}(X_i), Y_i), \ i \in \mathcal{D}_{\text{train}}$
where $\hat\mu_{-\mathcal{D}_{k(i)}}(X_i)$ denotes the CV prediction for $X_i$. In other words, for each CV fold $k=1,...,K$ and each training instance $i=1,...,n$ the model is trained on all training data excluding the fold containing $i$. The fitted model is then used to predict out-of-sample from $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $\hat\mu_{-\mathcal{D}_{k(i)}}(X_i)$ and the true value $Y_i$.
MLJModelInterface.fit
— MethodMMI.fit(conf_model::CVPlusRegressor, verbosity, X, y)
For the CVPlusRegressor
nonconformity scores are computed though cross-validation (CV) as follows,
$S_i^{\text{CV}} = s(X_i, Y_i) = h(\hat\mu_{-\mathcal{D}_{k(i)}}(X_i), Y_i), \ i \in \mathcal{D}_{\text{train}}$
where $\hat\mu_{-\mathcal{D}_{k(i)}}(X_i)$ denotes the CV prediction for $X_i$. In other words, for each CV fold $k=1,...,K$ and each training instance $i=1,...,n$ the model is trained on all training data excluding the fold containing $i$. The fitted model is then used to predict out-of-sample from $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $\hat\mu_{-\mathcal{D}_{k(i)}}(X_i)$ and the true value $Y_i$.
MLJModelInterface.fit
— MethodMMI.fit(conf_model::ConformalQuantileRegressor, verbosity, X, y)
For the ConformalQuantileRegressor
nonconformity scores are computed as follows:
$S_i^{\text{CAL}} = s(X_i, Y_i) = h(\hat\mu_{\alpha_{lo}}(X_i), \hat\mu_{\alpha_{hi}}(X_i) ,Y_i), \ i \in \mathcal{D}_{\text{calibration}}$
A typical choice for the heuristic function is $h(\hat\mu_{\alpha_{lo}}(X_i), \hat\mu_{\alpha_{hi}}(X_i) ,Y_i)= max\{\hat\mu_{\alpha_{low}}(X_i)-Y_i, Y_i-\hat\mu_{\alpha_{hi}}(X_i)\}$ where $\hat\mu$ denotes the model fitted on training data $\mathcal{D}_{\text{train}}` and$\alpha{lo}, \alpha{hi}`` lower and higher percentile.
MLJModelInterface.fit
— MethodMMI.fit(conf_model::JackknifeMinMaxRegressor, verbosity, X, y)
For the JackknifeMinMaxRegressor
nonconformity scores are computed in the same way as for the JackknifeRegressor
. Specifically, we have,
$S_i^{\text{LOO}} = s(X_i, Y_i) = h(\hat\mu_{-i}(X_i), Y_i), \ i \in \mathcal{D}_{\text{train}}$
where $\hat\mu_{-i}(X_i)$ denotes the leave-one-out prediction for $X_i$. In other words, for each training instance $i=1,...,n$ the model is trained on all training data excluding $i$. The fitted model is then used to predict out-of-sample from $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $\hat\mu_{-i}(X_i)$ and the true value $Y_i$.
MLJModelInterface.fit
— MethodMMI.fit(conf_model::JackknifePlusMinMaxAbRegressor, verbosity, X, y)
For the JackknifePlusAbMinMaxRegressor
nonconformity scores are as,
$S_i^{\text{J+MinMax}} = s(X_i, Y_i) = h(agg(\hat\mu_{B_{K(-i)}}(X_i)), Y_i), \ i \in \mathcal{D}_{\text{train}}$
where $agg(\hat\mu_{B_{K(-i)}}(X_i))$ denotes the aggregate predictions, typically mean or median, for each $X_i$ (with $K_{-i}$ the bootstraps not containing $X_i$). In other words, B models are trained on boostrapped sampling, the fitted models are then used to create aggregated prediction of out-of-sample $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $agg(\hat\mu_{B_{K(-i)}}(X_i))$ and the true value $Y_i$.
MLJModelInterface.fit
— MethodMMI.fit(conf_model::JackknifePlusAbRegressor, verbosity, X, y)
For the JackknifePlusAbRegressor
nonconformity scores are computed as
$$ S_i^{\text{J+ab}} = s(X_i, Y_i) = h(agg(\hat\mu_{B_{K(-i)}}(X_i)), Y_i), \ i \in \mathcal{D}_{\text{train}} $$
where $agg(\hat\mu_{B_{K(-i)}}(X_i))$ denotes the aggregate predictions, typically mean or median, for each $X_i$ (with $K_{-i}$ the bootstraps not containing $X_i$). In other words, B models are trained on boostrapped sampling, the fitted models are then used to create aggregated prediction of out-of-sample $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $agg(\hat\mu_{B_{K(-i)}}(X_i))$ and the true value $Y_i$.
MLJModelInterface.fit
— MethodMMI.fit(conf_model::JackknifePlusRegressor, verbosity, X, y)
For the JackknifePlusRegressor
nonconformity scores are computed in the same way as for the JackknifeRegressor
. Specifically, we have,
$S_i^{\text{LOO}} = s(X_i, Y_i) = h(\hat\mu_{-i}(X_i), Y_i), \ i \in \mathcal{D}_{\text{train}}$
where $\hat\mu_{-i}(X_i)$ denotes the leave-one-out prediction for $X_i$. In other words, for each training instance $i=1,...,n$ the model is trained on all training data excluding $i$. The fitted model is then used to predict out-of-sample from $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $\hat\mu_{-i}(X_i)$ and the true value $Y_i$.
MLJModelInterface.fit
— MethodMMI.fit(conf_model::JackknifeRegressor, verbosity, X, y)
For the JackknifeRegressor
nonconformity scores are computed through a leave-one-out (LOO) procedure as follows,
$S_i^{\text{LOO}} = s(X_i, Y_i) = h(\hat\mu_{-i}(X_i), Y_i), \ i \in \mathcal{D}_{\text{train}}$
where $\hat\mu_{-i}(X_i)$ denotes the leave-one-out prediction for $X_i$. In other words, for each training instance $i=1,...,n$ the model is trained on all training data excluding $i$. The fitted model is then used to predict out-of-sample from $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $\hat\mu_{-i}(X_i)$ and the true value $Y_i$.
MLJModelInterface.fit
— MethodMMI.fit(conf_model::NaiveClassifier, verbosity, X, y)
For the NaiveClassifier
nonconformity scores are computed in-sample as follows:
$S_i^{\text{IS}} = s(X_i, Y_i) = h(\hat\mu(X_i), Y_i), \ i \in \mathcal{D}_{\text{calibration}}$
A typical choice for the heuristic function is $h(\hat\mu(X_i), Y_i)=1-\hat\mu(X_i)_{Y_i}$ where $\hat\mu(X_i)_{Y_i}$ denotes the softmax output of the true class and $\hat\mu$ denotes the model fitted on training data $\mathcal{D}_{\text{train}}$.
MLJModelInterface.fit
— MethodMMI.fit(conf_model::NaiveRegressor, verbosity, X, y)
For the NaiveRegressor
nonconformity scores are computed in-sample as follows:
$S_i^{\text{IS}} = s(X_i, Y_i) = h(\hat\mu(X_i), Y_i), \ i \in \mathcal{D}_{\text{train}}$
A typical choice for the heuristic function is $h(\hat\mu(X_i),Y_i)=|Y_i-\hat\mu(X_i)|$ where $\hat\mu$ denotes the model fitted on training data $\mathcal{D}_{\text{train}}$.
MLJModelInterface.fit
— MethodMMI.fit(conf_model::SimpleInductiveClassifier, verbosity, X, y)
For the SimpleInductiveClassifier
nonconformity scores are computed as follows:
$S_i^{\text{CAL}} = s(X_i, Y_i) = h(\hat\mu(X_i), Y_i), \ i \in \mathcal{D}_{\text{calibration}}$
A typical choice for the heuristic function is $h(\hat\mu(X_i), Y_i)=1-\hat\mu(X_i)_{Y_i}$ where $\hat\mu(X_i)_{Y_i}$ denotes the softmax output of the true class and $\hat\mu$ denotes the model fitted on training data $\mathcal{D}_{\text{train}}$. The simple approach only takes the softmax probability of the true label into account.
MLJModelInterface.fit
— MethodMMI.fit(conf_model::SimpleInductiveRegressor, verbosity, X, y)
For the SimpleInductiveRegressor
nonconformity scores are computed as follows:
$S_i^{\text{CAL}} = s(X_i, Y_i) = h(\hat\mu(X_i), Y_i), \ i \in \mathcal{D}_{\text{calibration}}$
A typical choice for the heuristic function is $h(\hat\mu(X_i),Y_i)=|Y_i-\hat\mu(X_i)|$ where $\hat\mu$ denotes the model fitted on training data $\mathcal{D}_{\text{train}}$.
MLJModelInterface.fit
— MethodMMI.fit(conf_model::TimeSeriesRegressorEnsembleBatch, verbosity, X, y)
For the TimeSeriesRegressorEnsembleBatch
nonconformity scores are computed as
$$ S_i^{\text{J+ab}} = s(X_i, Y_i) = h(agg(\hat\mu_{B_{K(-i)}}(X_i)), Y_i), \ i \in \mathcal{D}_{\text{train}} $$
where $agg(\hat\mu_{B_{K(-i)}}(X_i))$ denotes the aggregate predictions, typically mean or median, for each $X_i$ (with $K_{-i}$ the bootstraps not containing $X_i$). In other words, B models are trained on boostrapped sampling, the fitted models are then used to create aggregated prediction of out-of-sample $X_i$. The corresponding nonconformity score is then computed by applying a heuristic uncertainty measure $h(\cdot)$ to the fitted value $agg(\hat\mu_{B_{K(-i)}}(X_i))$ and the true value $Y_i$.
MLJModelInterface.predict
— MethodMMI.predict(conf_model::AdaptiveInductiveClassifier, fitresult, Xnew)
For the AdaptiveInductiveClassifier
prediction sets are computed as follows,
$\hat{C}_{n,\alpha}(X_{n+1}) = \left\{y: s(X_{n+1},y) \le \hat{q}_{n, \alpha}^{+} \{S_i^{\text{CAL}}\} \right\}, i \in \mathcal{D}_{\text{calibration}}$
where $\mathcal{D}_{\text{calibration}}$ denotes the designated calibration data.
MLJModelInterface.predict
— MethodMMI.predict(conf_model::CVMinMaxRegressor, fitresult, Xnew)
For the CVMinMaxRegressor
prediction intervals are computed as follows,
$\hat{C}_{n,\alpha}(X_{n+1}) = \left[ \min_{i=1,...,n} \hat\mu_{-\mathcal{D}_{k(i)}}(X_{n+1}) - \hat{q}_{n, \alpha}^{+} \{S_i^{\text{CV}} \}, \max_{i=1,...,n} \hat\mu_{-\mathcal{D}_{k(i)}}(X_{n+1}) + \hat{q}_{n, \alpha}^{+} \{ S_i^{\text{CV}}\} \right] , i \in \mathcal{D}_{\text{train}}$
where $\hat\mu_{-\mathcal{D}_{k(i)}}$ denotes the model fitted on training data with subset $\mathcal{D}_{k(i)}$ that contains the $i$ th point removed.
MLJModelInterface.predict
— MethodMMI.predict(conf_model::CVPlusRegressor, fitresult, Xnew)
For the CVPlusRegressor
prediction intervals are computed in much same way as for the JackknifePlusRegressor
. Specifically, we have,
$\hat{C}_{n,\alpha}(X_{n+1}) = \left[ \hat{q}_{n, \alpha}^{-} \{\hat\mu_{-\mathcal{D}_{k(i)}}(X_{n+1}) - S_i^{\text{CV}} \}, \hat{q}_{n, \alpha}^{+} \{\hat\mu_{-\mathcal{D}_{k(i)}}(X_{n+1}) + S_i^{\text{CV}}\} \right] , \ i \in \mathcal{D}_{\text{train}}$
where $\hat\mu_{-\mathcal{D}_{k(i)}}$ denotes the model fitted on training data with fold $\mathcal{D}_{k(i)}$ that contains the $i$ th point removed.
The JackknifePlusRegressor
is a special case of the CVPlusRegressor
for which $K=n$.
MLJModelInterface.predict
— MethodMMI.predict(conf_model::ConformalQuantileRegressor, fitresult, Xnew)
For the ConformalQuantileRegressor
prediction intervals are computed as follows,
$\hat{C}_{n,\alpha}(X_{n+1}) = [\hat\mu_{\alpha_{lo}}(X_{n+1}) - \hat{q}_{n, \alpha} \{S_i^{\text{CAL}} \}, \hat\mu_{\alpha_{hi}}(X_{n+1}) + \hat{q}_{n, \alpha} \{S_i^{\text{CAL}} \}], \ i \in \mathcal{D}_{\text{calibration}}$
where $\mathcal{D}_{\text{calibration}}$ denotes the designated calibration data.
MLJModelInterface.predict
— MethodMMI.predict(conf_model::JackknifeMinMaxRegressor, fitresult, Xnew)
For the JackknifeMinMaxRegressor
prediction intervals are computed as follows,
$\hat{C}_{n,\alpha}(X_{n+1}) = \left[ \min_{i=1,...,n} \hat\mu_{-i}(X_{n+1}) - \hat{q}_{n, \alpha}^{+} \{S_i^{\text{LOO}} \}, \max_{i=1,...,n} \hat\mu_{-i}(X_{n+1}) + \hat{q}_{n, \alpha}^{+} \{S_i^{\text{LOO}}\} \right] , i \in \mathcal{D}_{\text{train}}$
where $\hat\mu_{-i}$ denotes the model fitted on training data with $i$th point removed. The jackknife-minmax procedure is more conservative than the JackknifePlusRegressor
.
MLJModelInterface.predict
— MethodMMI.predict(conf_model::JackknifePlusAbMinMaxRegressor, fitresult, Xnew)
For the JackknifePlusAbMinMaxRegressor
prediction intervals are computed as follows,
$\hat{C}_{n,\alpha}^{J+MinMax}(X_{n+1}) = \left[ \min_{i=1,...,n} \hat\mu_{-i}(X_{n+1}) - \hat{q}_{n, \alpha}^{+} \{S_i^{\text{J+MinMax}} \}, \max_{i=1,...,n} \hat\mu_{-i}(X_{n+1}) + \hat{q}_{n, \alpha}^{+} \{S_i^{\text{J+MinMax}}\} \right] , i \in \mathcal{D}_{\text{train}}$
where $\hat\mu_{-i}$ denotes the model fitted on training data with $i$th point removed. The jackknife+ab-minmax procedure is more conservative than the JackknifePlusAbRegressor
.
MLJModelInterface.predict
— MethodMMI.predict(conf_model::JackknifePlusAbRegressor, fitresult, Xnew)
For the JackknifePlusAbRegressor
prediction intervals are computed as follows,
$\hat{C}_{n,\alpha, B}^{J+ab}(X_{n+1}) = \left[ \hat{q}_{n, \alpha}^{-} \{\hat\mu_{agg(-i)}(X_{n+1}) - S_i^{\text{J+ab}} \}, \hat{q}_{n, \alpha}^{+} \{\hat\mu_{agg(-i)}(X_{n+1}) + S_i^{\text{J+ab}}\} \right] , i \in \mathcal{D}_{\text{train}}$
where $\hat\mu_{agg(-i)}$ denotes the aggregated models $\hat\mu_{1}, ...., \hat\mu_{B}$ fitted on bootstrapped data (B) does not include the $i$th data point. The jackknife$+$ procedure is more stable than the JackknifeRegressor
.
MLJModelInterface.predict
— MethodMMI.predict(conf_model::JackknifePlusRegressor, fitresult, Xnew)
For the JackknifePlusRegressor
prediction intervals are computed as follows,
$\hat{C}_{n,\alpha}(X_{n+1}) = \left[ \hat{q}_{n, \alpha}^{-} \{\hat\mu_{-i}(X_{n+1}) - S_i^{\text{LOO}} \}, \hat{q}_{n, \alpha}^{+} \{\hat\mu_{-i}(X_{n+1}) + S_i^{\text{LOO}}\} \right] , i \in \mathcal{D}_{\text{train}}$
where $\hat\mu_{-i}$ denotes the model fitted on training data with $i$th point removed. The jackknife$+$ procedure is more stable than the JackknifeRegressor
.
MLJModelInterface.predict
— MethodMMI.predict(conf_model::JackknifeRegressor, fitresult, Xnew)
For the JackknifeRegressor
prediction intervals are computed as follows,
$\hat{C}_{n,\alpha}(X_{n+1}) = \hat\mu(X_{n+1}) \pm \hat{q}_{n, \alpha}^{+} \{S_i^{\text{LOO}}\}, \ i \in \mathcal{D}_{\text{train}}$
where $S_i^{\text{LOO}}$ denotes the nonconformity that is generated as explained in fit(conf_model::JackknifeRegressor, verbosity, X, y)
. The jackknife procedure addresses the overfitting issue associated with the NaiveRegressor
.
MLJModelInterface.predict
— MethodMMI.predict(conf_model::NaiveClassifier, fitresult, Xnew)
For the NaiveClassifier
prediction sets are computed as follows:
$\hat{C}_{n,\alpha}(X_{n+1}) = \left\{y: s(X_{n+1},y) \le \hat{q}_{n, \alpha}^{+} \{S_i^{\text{IS}} \} \right\}, \ i \in \mathcal{D}_{\text{train}}$
The naive approach typically produces prediction regions that undercover due to overfitting.
MLJModelInterface.predict
— MethodMMI.predict(conf_model::NaiveRegressor, fitresult, Xnew)
For the NaiveRegressor
prediction intervals are computed as follows:
$\hat{C}_{n,\alpha}(X_{n+1}) = \hat\mu(X_{n+1}) \pm \hat{q}_{n, \alpha}^{+} \{S_i^{\text{IS}} \}, \ i \in \mathcal{D}_{\text{train}}$
The naive approach typically produces prediction regions that undercover due to overfitting.
MLJModelInterface.predict
— MethodMMI.predict(conf_model::SimpleInductiveClassifier, fitresult, Xnew)
For the SimpleInductiveClassifier
prediction sets are computed as follows,
$\hat{C}_{n,\alpha}(X_{n+1}) = \left\{y: s(X_{n+1},y) \le \hat{q}_{n, \alpha}^{+} \{S_i^{\text{CAL}}\} \right\}, \ i \in \mathcal{D}_{\text{calibration}}$
where $\mathcal{D}_{\text{calibration}}$ denotes the designated calibration data.
MLJModelInterface.predict
— MethodMMI.predict(conf_model::SimpleInductiveRegressor, fitresult, Xnew)
For the SimpleInductiveRegressor
prediction intervals are computed as follows,
$\hat{C}_{n,\alpha}(X_{n+1}) = \hat\mu(X_{n+1}) \pm \hat{q}_{n, \alpha}^{+} \{S_i^{\text{CAL}} \}, \ i \in \mathcal{D}_{\text{calibration}}$
where $\mathcal{D}_{\text{calibration}}$ denotes the designated calibration data.
MLJModelInterface.predict
— MethodMMI.predict(conf_model::TimeSeriesRegressorEnsembleBatch, fitresult, Xnew)
For the TimeSeriesRegressorEnsembleBatch
prediction intervals are computed as follows,
$\hat{C}_{n,\alpha, B}^{J+ab}(X_{n+1}) = \left[ \hat{q}_{n, \alpha}^{-} \{\hat\mu_{agg(-i)}(X_{n+1}) - S_i^{\text{J+ab}} \}, \hat{q}_{n, \alpha}^{+} \{\hat\mu_{agg(-i)}(X_{n+1}) + S_i^{\text{J+ab}}\} \right] , i \in \mathcal{D}_{\text{train}}$
where $\hat\mu_{agg(-i)}$ denotes the aggregated models $\hat\mu_{1}, ...., \hat\mu_{B}$ fitted on bootstrapped data (B) does not include the $i$th data point. The jackknife$+$ procedure is more stable than the JackknifeRegressor
.
ConformalPrediction.ConformalTraining.ConformalNNClassifier
— TypeThe ConformalNNClassifier
struct is a wrapper for a ConformalModel
that can be used with MLJFlux.jl.
ConformalPrediction.ConformalTraining.ConformalNNRegressor
— TypeThe ConformalNNRegressor
struct is a wrapper for a ConformalModel
that can be used with MLJFlux.jl.
ConformalPrediction.ConformalTraining.classification_loss
— Methodclassification_loss(
conf_model::ConformalProbabilisticSet, fitresult, X, y;
loss_matrix::Union{AbstractMatrix,UniformScaling}=UniformScaling(1.0),
temp::Real=0.1
)
Computes the calibration loss following Stutz et al. (2022): https://openreview.net/pdf?id=t8O-4LKFVx. Following the notation in the paper, the loss is computed as,
\[\mathcal{L}(C_{\theta}(x;\tau),y) = \sum_k L_{y,k} \left[ (1 - C_{\theta,k}(x;\tau)) \mathbf{I}_{y=k} + C_{\theta,k}(x;\tau) \mathbf{I}_{y\ne k} \right]\]
where $\tau$ is just the quantile q̂
and $\kappa$ is the target set size (defaults to $1$).
ConformalPrediction.ConformalTraining.qminus_smooth
— Functionqminus_smooth(v::AbstractArray, coverage::AbstractFloat=0.9)
Implements the $\hat{q}_{n,\alpha}^{-}$ finite-sample corrected quantile function as defined in Barber et al. (2020): https://arxiv.org/pdf/1905.02928.pdf. To allow for differentiability, we use the soft sort function from InferOpt.jl.
ConformalPrediction.ConformalTraining.qplus_smooth
— Functionqplus_smooth(v::AbstractArray, coverage::AbstractFloat=0.9)
Implements the $\hat{q}_{n,\alpha}^{+}$ finite-sample corrected quantile function as defined in Barber et al. (2020): https://arxiv.org/pdf/1905.02928.pdf. To allow for differentiability, we use the soft sort function from InferOpt.jl.
ConformalPrediction.ConformalTraining.score
— FunctionConformalPrediction.score(conf_model::AdaptiveInductiveClassifier, ::Type{<:EitherEnsembleModel{<:MLJFluxModel}}, fitresult, X, y::Union{Nothing,AbstractArray}=nothing)
Overloads the score
function for ensembles of MLJFluxModel
types.
ConformalPrediction.ConformalTraining.score
— FunctionConformalPrediction.score(conf_model::AdaptiveInductiveClassifier, ::Type{<:MLJFluxModel}, fitresult, X, y::Union{Nothing,AbstractArray}=nothing)
Overloads the score
function for the MLJFluxModel
type.
ConformalPrediction.ConformalTraining.smooth_size_loss
— Methodfunction smooth_size_loss(
conf_model::ConformalProbabilisticSet, fitresult, X;
temp::Real=0.1, κ::Real=1.0
)
Computes the smooth (differentiable) size loss following Stutz et al. (2022): https://openreview.net/pdf?id=t8O-4LKFVx. First, soft assignment probabilities are computed for new data X
. Then (following the notation in the paper) the loss is computed as,
\[\Omega(C_{\theta}(x;\tau)) = \max (0, \sum_k C_{\theta,k}(x;\tau) - \kappa)\]
where $\tau$ is just the quantile q̂
and $\kappa$ is the target set size (defaults to $1$). For empty sets, the loss is computed as $K - \kappa$, that is the maximum set size minus the target set size.
ConformalPrediction.ConformalTraining.soft_assignment
— Methodsoft_assignment(conf_model::ConformalProbabilisticSet, fitresult, X; temp::Real=0.1)
This function can be used to compute soft assigment probabilities for new data X
as in soft_assignment(conf_model::ConformalProbabilisticSet; temp::Real=0.1)
. When a fitted model $\mu$ (fitresult
) and new samples X
are supplied, non-conformity scores are first computed for the new data points. Then the existing threshold/quantile q̂
is used to compute the final soft assignments.
ConformalPrediction.ConformalTraining.soft_assignment
— Methodsoft_assignment(conf_model::ConformalProbabilisticSet; temp::Real=0.1)
Computes soft assignment scores for each label and sample. That is, the probability of label k
being included in the confidence set. This implementation follows Stutz et al. (2022): https://openreview.net/pdf?id=t8O-4LKFVx. Contrary to the paper, we use non-conformity scores instead of conformity scores, hence the sign swap.
ConformalPrediction.score
— FunctionConformalPrediction.score(conf_model::InductiveModel, model::MLJFluxModel, fitresult, X, y::Union{Nothing,AbstractArray}=nothing)
Overloads the score
function for the MLJFluxModel
type.
ConformalPrediction.score
— FunctionConformalPrediction.score(conf_model::SimpleInductiveClassifier, ::Type{<:EitherEnsembleModel{<:MLJFluxModel}}, fitresult, X, y::Union{Nothing,AbstractArray}=nothing)
Overloads the score
function for ensembles of MLJFluxModel
types.
MLJFlux.shape
— Methodshape(model::NeuralNetworkRegressor, X, y)
A private method that returns the shape of the input and output of the model for given data X
and y
.
MLJFlux.train!
— MethodMLJFlux.train!(model::ConformalNN, penalty, chain, optimiser, X, y)
Implements the conformal traning procedure for the ConformalNN
type.