RandomFeatures
Kernel and Covariance structure
CalibrateEmulateSample.Emulators.OneDimFactor
— Typestruct OneDimFactor <: CalibrateEmulateSample.Emulators.CovarianceStructureType
covariance structure for a one-dimensional space
CalibrateEmulateSample.Emulators.DiagonalFactor
— Typestruct DiagonalFactor{FT<:AbstractFloat} <: CalibrateEmulateSample.Emulators.CovarianceStructureType
builds a diagonal covariance structure
CalibrateEmulateSample.Emulators.CholeskyFactor
— Typestruct CholeskyFactor{FT<:AbstractFloat} <: CalibrateEmulateSample.Emulators.CovarianceStructureType
builds a general positive-definite covariance structure
CalibrateEmulateSample.Emulators.LowRankFactor
— Typestruct LowRankFactor{FT<:AbstractFloat} <: CalibrateEmulateSample.Emulators.CovarianceStructureType
builds a covariance structure that deviates from the identity with a low-rank perturbation. This perturbation is diagonalized in the low-rank space
CalibrateEmulateSample.Emulators.HierarchicalLowRankFactor
— Typestruct HierarchicalLowRankFactor{FT<:AbstractFloat} <: CalibrateEmulateSample.Emulators.CovarianceStructureType
builds a covariance structure that deviates from the identity with a more general low-rank perturbation
CalibrateEmulateSample.Emulators.SeparableKernel
— Typestruct SeparableKernel{CST1<:CalibrateEmulateSample.Emulators.CovarianceStructureType, CST2<:CalibrateEmulateSample.Emulators.CovarianceStructureType} <: CalibrateEmulateSample.Emulators.KernelStructureType
Builds a separable kernel, i.e. one that accounts for input and output covariance structure separately
CalibrateEmulateSample.Emulators.NonseparableKernel
— Typestruct NonseparableKernel{CST<:CalibrateEmulateSample.Emulators.CovarianceStructureType} <: CalibrateEmulateSample.Emulators.KernelStructureType
Builds a nonseparable kernel, i.e. one that accounts for a joint input and output covariance structure
CalibrateEmulateSample.Emulators.calculate_n_hyperparameters
— Functioncalculate_n_hyperparameters(
d::Int64,
odf::CalibrateEmulateSample.Emulators.OneDimFactor
) -> Int64
calculates the number of hyperparameters generated by the choice of covariance structure
CalibrateEmulateSample.Emulators.hyperparameters_from_flat
— Functionhyperparameters_from_flat(
x::AbstractVector,
odf::CalibrateEmulateSample.Emulators.OneDimFactor
)
reshapes a list of hyperparameters into a covariance matrix based on the selected structure
CalibrateEmulateSample.Emulators.build_default_prior
— Functionbuild_default_prior(
name::AbstractString,
n_hp::Int64,
odf::CalibrateEmulateSample.Emulators.OneDimFactor
)
builds a prior distribution for the kernel hyperparameters to initialize optimization.
Scalar interface
CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface
— Typestruct ScalarRandomFeatureInterface{S<:AbstractString, RNG<:Random.AbstractRNG, KST<:CalibrateEmulateSample.Emulators.KernelStructureType} <: CalibrateEmulateSample.Emulators.RandomFeatureInterface
Structure holding the Scalar Random Feature models.
Fields
rfms::Vector{RandomFeatures.Methods.RandomFeatureMethod}
: vector ofRandomFeatureMethod
s, contains the feature structure, batch-sizes and regularizationfitted_features::Vector{RandomFeatures.Methods.Fit}
: vector ofFit
s, containing the matrix decomposition and coefficients of RF when fitted to databatch_sizes::Union{Nothing, Dict{S, Int64}} where S<:AbstractString
: batch sizesn_features::Union{Nothing, Int64}
: n_featuresinput_dim::Int64
: input dimensionrng::Random.AbstractRNG
: choice of random number generatorkernel_structure::CalibrateEmulateSample.Emulators.KernelStructureType
: Kernel structure type (e.g. Separable or Nonseparable)feature_decomposition::AbstractString
: Random Feature decomposition, choose from "svd" or "cholesky" (default)optimizer_options::Dict{S} where S<:AbstractString
: dictionary of options for hyperparameter optimizeroptimizer::Vector
: diagnostics from optimizer
CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface
— MethodScalarRandomFeatureInterface(
n_features::Int64,
input_dim::Int64;
kernel_structure,
batch_sizes,
rng,
feature_decomposition,
optimizer_options
) -> CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface{String, Random._GLOBAL_RNG, CalibrateEmulateSample.Emulators.SeparableKernel{CST1, CalibrateEmulateSample.Emulators.OneDimFactor}} where CST1<:CalibrateEmulateSample.Emulators.CovarianceStructureType
Constructs a ScalarRandomFeatureInterface <: MachineLearningTool
interface for the RandomFeatures.jl
package for multi-input and single- (or decorrelated-)output emulators.
n_features
- the number of random featuresinput_dim
- the dimension of the input spacekernel_structure
- - a prescribed form of kernel structurebatch_sizes = nothing
- Dictionary of batch sizes passedRandomFeatures.jl
object (see definition there)rng = Random.GLOBAL_RNG
- random number generatorfeature_decomposition = "cholesky"
- choice of how to store decompositions of random features,cholesky
orsvd
availableoptimizer_options = nothing
- Dict of options to pass into EKI optimization of hyperparameters (defaults created inScalarRandomFeatureInterface
constructor):- "prior": the prior for the hyperparameter optimization
- "priorinscale": use this to tune the input prior scale
- "n_ensemble": number of ensemble members
- "n_iteration": number of eki iterations
- "covsamplemultiplier": increase for more samples to estimate covariance matrix in optimization (default 10.0, minimum 0.0)
- "scheduler": Learning rate Scheduler (a.k.a. EKP timestepper) Default: DataMisfitController
- "tikhonov": tikhonov regularization parameter if >0
- "inflation": additive inflation ∈ [0,1] with 0 being no inflation
- "train_fraction": e.g. 0.8 (default) means 80:20 train - test split
- "nfeaturesopt": fix the number of features for optimization (default
n_features
, as used for prediction) - "multithread": how to multithread. "ensemble" (default) threads across ensemble members "tullio" threads random feature matrix algebra
- "accelerator": use EKP accelerators (default is no acceleration)
- "verbose" => false, verbose optimizer statements
- "cov_correction" => "shrinkage", type of conditioning to improve estimated covariance (Ledoit Wolfe 03), also "nice" for (Vishny, Morzfeld et al. 2024)
CalibrateEmulateSample.Emulators.build_models!
— Methodbuild_models!(
srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface,
input_output_pairs::EnsembleKalmanProcesses.DataContainers.PairedDataContainer{FT<:AbstractFloat}
) -> Union{Nothing, Vector}
Builds the random feature method from hyperparameters. We use cosine activation functions and a Multivariate Normal distribution (from Distributions.jl
) with mean M=0, and input covariance U built with the CovarianceStructureType
.
GaussianProcesses.predict
— Methodpredict(
srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface,
new_inputs::AbstractMatrix;
multithread
) -> Tuple{Any, Any}
Prediction of data observation (not latent function) at new inputs (passed in as columns in a matrix). That is, we add the observational noise into predictions.
Vector Interface
CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
— Typestruct VectorRandomFeatureInterface{S<:AbstractString, RNG<:Random.AbstractRNG, KST<:CalibrateEmulateSample.Emulators.KernelStructureType} <: CalibrateEmulateSample.Emulators.RandomFeatureInterface
Structure holding the Vector Random Feature models.
Fields
rfms::Vector{RandomFeatures.Methods.RandomFeatureMethod}
: A vector ofRandomFeatureMethod
s, contains the feature structure, batch-sizes and regularizationfitted_features::Vector{RandomFeatures.Methods.Fit}
: vector ofFit
s, containing the matrix decomposition and coefficients of RF when fitted to databatch_sizes::Union{Nothing, Dict{S, Int64}} where S<:AbstractString
: batch sizesn_features::Union{Nothing, Int64}
: number of featuresinput_dim::Int64
: input dimensionoutput_dim::Int64
: output_dimensionrng::Random.AbstractRNG
: rngregularization::Vector{Union{LinearAlgebra.Diagonal, LinearAlgebra.UniformScaling, Matrix}}
: regularizationkernel_structure::CalibrateEmulateSample.Emulators.KernelStructureType
: Kernel structure type (e.g. Separable or Nonseparable)feature_decomposition::AbstractString
: Random Feature decomposition, choose from "svd" or "cholesky" (default)optimizer_options::Dict
: dictionary of options for hyperparameter optimizeroptimizer::Vector
: diagnostics from optimizer
CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
— MethodVectorRandomFeatureInterface(
n_features::Int64,
input_dim::Int64,
output_dim::Int64;
kernel_structure,
batch_sizes,
rng,
feature_decomposition,
optimizer_options
) -> CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface{String, Random._GLOBAL_RNG, CalibrateEmulateSample.Emulators.SeparableKernel{CST1, CST2}} where {CST1<:CalibrateEmulateSample.Emulators.CovarianceStructureType, CST2<:CalibrateEmulateSample.Emulators.CovarianceStructureType}
Constructs a VectorRandomFeatureInterface <: MachineLearningTool
interface for the RandomFeatures.jl
package for multi-input and multi-output emulators.
n_features
- the number of random featuresinput_dim
- the dimension of the input spaceoutput_dim
- the dimension of the output spacekernel_structure
- - a prescribed form of kernel structurebatch_sizes = nothing
- Dictionary of batch sizes passedRandomFeatures.jl
object (see definition there)rng = Random.GLOBAL_RNG
- random number generatorfeature_decomposition = "cholesky"
- choice of how to store decompositions of random features,cholesky
orsvd
availableoptimizer_options = nothing
- Dict of options to pass into EKI optimization of hyperparameters (defaults created inVectorRandomFeatureInterface
constructor):- "prior": the prior for the hyperparameter optimization
- "priorinscale"/"prioroutscale": use these to tune the input/output prior scale.
- "n_ensemble": number of ensemble members
- "n_iteration": number of eki iterations
- "scheduler": Learning rate Scheduler (a.k.a. EKP timestepper) Default: DataMisfitController
- "covsamplemultiplier": increase for more samples to estimate covariance matrix in optimization (default 10.0, minimum 0.0)
- "tikhonov": tikhonov regularization parameter if > 0
- "inflation": additive inflation ∈ [0,1] with 0 being no inflation
- "train_fraction": e.g. 0.8 (default) means 80:20 train - test split
- "nfeaturesopt": fix the number of features for optimization (default
n_features
, as used for prediction) - "multithread": how to multithread. "ensemble" (default) threads across ensemble members "tullio" threads random feature matrix algebra
- "accelerator": use EKP accelerators (default is no acceleration)
- "verbose" => false, verbose optimizer statements to check convergence, priors and optimal parameters.
- "cov_correction" => "shrinkage", type of conditioning to improve estimated covariance (Ledoit Wolfe 03), also "nice" for (Vishny, Morzfeld et al. 2024)
CalibrateEmulateSample.Emulators.build_models!
— Methodbuild_models!(
vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface,
input_output_pairs::EnsembleKalmanProcesses.DataContainers.PairedDataContainer{FT<:AbstractFloat};
regularization_matrix
) -> Union{Nothing, Vector{Union{LinearAlgebra.Diagonal, LinearAlgebra.UniformScaling, Matrix}}}
Build Vector Random Feature model for the input-output pairs subject to regularization, and optimizes the hyperparameters with EKP.
GaussianProcesses.predict
— Methodpredict(
vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface,
new_inputs::AbstractMatrix
) -> Tuple{Any, Any}
Prediction of data observation (not latent function) at new inputs (passed in as columns in a matrix). That is, we add the observational noise into predictions.
Other utilities
CalibrateEmulateSample.Emulators.get_rfms
— Functionget_rfms(
srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface
) -> Vector{RandomFeatures.Methods.RandomFeatureMethod}
gets the rfms field
get_rfms(
vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
) -> Vector{RandomFeatures.Methods.RandomFeatureMethod}
Gets the rfms field
CalibrateEmulateSample.Emulators.get_fitted_features
— Functionget_fitted_features(
srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface
) -> Vector{RandomFeatures.Methods.Fit}
gets the fitted_features field
get_fitted_features(
vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
) -> Vector{RandomFeatures.Methods.Fit}
Gets the fitted_features field
CalibrateEmulateSample.Emulators.get_batch_sizes
— Functionget_batch_sizes(
srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface
) -> Union{Nothing, Dict{S, Int64}} where S<:AbstractString
gets batch_sizes the field
get_batch_sizes(
vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
) -> Union{Nothing, Dict{S, Int64}} where S<:AbstractString
Gets the batch_sizes field
CalibrateEmulateSample.Emulators.get_n_features
— Functionget_n_features(
srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface
) -> Union{Nothing, Int64}
gets the n_features field
get_n_features(
vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
) -> Union{Nothing, Int64}
Gets the n_features field
CalibrateEmulateSample.Emulators.get_input_dim
— Functionget_input_dim(
srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface
) -> Int64
gets the input_dim field
get_input_dim(
vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
) -> Int64
Gets the input_dim field
CalibrateEmulateSample.Emulators.get_output_dim
— Functionget_output_dim(
vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
) -> Int64
Gets the output_dim field
EnsembleKalmanProcesses.get_rng
— Functionget_rng(
srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface
) -> Random.AbstractRNG
gets the rng field
get_rng(
vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
) -> Random.AbstractRNG
Gets the rng field
CalibrateEmulateSample.Emulators.get_kernel_structure
— Functionget_kernel_structure(
srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface
) -> CalibrateEmulateSample.Emulators.KernelStructureType
Gets the kernel_structure field
get_kernel_structure(
vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
) -> CalibrateEmulateSample.Emulators.KernelStructureType
Gets the kernel_structure field
CalibrateEmulateSample.Emulators.get_feature_decomposition
— Functionget_feature_decomposition(
srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface
) -> AbstractString
gets the feature_decomposition field
get_feature_decomposition(
vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
) -> AbstractString
Gets the feature_decomposition field
CalibrateEmulateSample.Emulators.get_optimizer_options
— Functionget_optimizer_options(
srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface
) -> Dict{S} where S<:AbstractString
gets the optimizer_options field
get_optimizer_options(
vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface
) -> Dict
Gets the optimizer_options field
CalibrateEmulateSample.Emulators.optimize_hyperparameters!
— Methodoptimize_hyperparameters!(
srfi::CalibrateEmulateSample.Emulators.ScalarRandomFeatureInterface,
args...;
kwargs...
)
Empty method, as optimization takes place within the build_models stage
CalibrateEmulateSample.Emulators.optimize_hyperparameters!
— Methodoptimize_hyperparameters!(
vrfi::CalibrateEmulateSample.Emulators.VectorRandomFeatureInterface,
args...;
kwargs...
)
Empty method, as optimization takes place within the build_models stage
CalibrateEmulateSample.Emulators.shrinkage_cov
— Functionshrinkage_cov(sample_mat::AbstractMatrix) -> Any
Calculate the empirical covariance, additionally applying a shrinkage operator (here the Ledoit Wolf 2004 shrinkage operation). Known to have better stability properties than Monte-Carlo for low sample sizes