This package exports

• BOpt, boptimize!
• acquisition types: ExpectedImprovement, ProbabilityOfImprovement, UpperConfidenceBound, ThompsonSamplingSimple, MutualInformation
• scaling of standard deviation in UpperConfidenceBound: BrochuBetaScaling, NoBetaScaling
• GP hyperparameter optimizer: MAPGPOptimizer, NoModelOptimizer
• Initializer: ScaledSobolIterator, ScaledLHSIterator
• optimization sense: Min, Max
• verbosity levels: Silent, Timings, Progress
• helper: maxduration!, maxiterations!

Use the REPL help, e.g. ?Bopt, to get more information.

BOpt(func, model, acquisition, modeloptimizer, lowerbounds, upperbounds;
          sense = Max, maxiterations = 10^4, maxduration = Inf,
          acquisitionoptions = NamedTuple(), repetitions = 1,
          verbosity = Progress,
          initializer_iterations = 5*length(lowerbounds),
          initializer = ScaledSobolIterator(lowerbounds, upperbounds,
                                            initializer_iterations))

Scales βt of UpperConfidenceBound as

βt = √(2 * log(t^(D/2 + 2) * π^2/(3δ)))

where t is the number of observations, D is the dimensionality of the input data points and δ is a small constant (default δ = 0.1).

See Brochu E., Cora V. M., de Freitas N. (2010), "A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning", https://arxiv.org/abs/1012.2599v1 page 16.

The expected improvement measures the expected improvement x - τ of a point x upon an incumbent target τ. For Gaussian distributions it is given by

(μ(x) - τ) * ϕ[(μ(x) - τ)/σ(x)] + σ(x) * Φ[(μ(x) - τ)/σ(x)]

where ϕ is the standard normal distribution function and Φ is the standard normal cumulative function, and μ(x), σ(x) are mean and standard deviation of the distribution at point x.

MAPGPOptimizer(; every = 10, kwargs...)

Set the GP hyperparameters to the maximum a posteriori (MAP) estimate every number of steps. Run BayesianOptimization.defaultoptions(MAPGPOptimizer) to see the default options. By default, all priors are flat. Uniform priors in an interval can be specified by setting the bounds of the form [lowerbound, upperbound], e.g. for a kernel with 3 parameters one would set kernbounds = [[-3, -3, -4], [2, 3, 1]]. Non-flat priors can be specified directly on the GP parameters, e.g. using Distributions; set_priors!(mean, [Normal(0., 1.)])

The mutual information measures the amount of information gained by querying at x. The parameter γ̂ gives a lower bound for the information on f from the queries {x}. For a Gaussian this is γ̂ = ∑σ²(x) and the mutual information at x is μ(x) + √(α)*(√(σ²(x)+γ̂) - √(γ̂))

where μ(x), σ(x) are mean and standard deviation of the distribution at point x.

See Contal E., Perchet V., Vayatis N. (2014), "Gaussian Process Optimization with Mutual Information" http://proceedings.mlr.press/v32/contal14.pdf

Applies no scaling to βt of UpperConfidenceBound.

Don't optimize the model ever.

The probability of improvement measures the probability that a point x leads to an improvement upon an incumbent target τ. For Gaussian distributions it is given by

Φ[(μ(x) - τ)/σ(x)]

where Φ is the standard normal cumulative distribution function and μ(x), σ(x) are mean and standard deviation of the distribution at point x.

ScaledSobolIterator(lowerbounds, upperbounds, N;
                    seq = SobolSeq(length(lowerbounds)))

Returns an iterator over N elements of a Sobol sequence between lowerbounds and upperbounds. The first N elements of the Sobol sequence are skipped for better uniformity (see https://github.com/stevengj/Sobol.jl)

The acquisition function associated with ThompsonSamplingSimple draws independent samples for each input x a function value from the model. Together with a gradient-free optimization method this leads to proposal points that might be similarly distributed as the maxima of true Thompson samples from GPs. True Thompson samples from a GP are simply functions from a GP. Maximizing these samples can be tricky, see e.g. http://hildobijl.com/Downloads/GPRT.pdf chapter 6.

For Gaussian distributions the upper confidence bound at x is given by μ(x) + βt * σ(x)

where βt is a fixed parameter in the case of NoBetaScaling or an observation size dependent parameter in the case of e.g. BrochuBetaScaling.

UpperConfidenceBound(; scaling = BrochuBetaScaling(.1), βt = 1)

 ScaledLHSIterator(lowerbounds, upperbounds, N)

Returns an iterator over N elements of a latin hyper cube sample between lowerbounds and upperbounds. See also ScaledSobolIterator for an iterator that has arguably better uniformity.

boptimize!(o::BOpt)

 maxduration!(s::IterationCounter, duration)
 maxduration!(o::BOpt, duration)

Sets the maximal duration per call of boptimize! to duration.

 maxiterations!(s::IterationCounter, N)
 maxiterations!(o::BOpt, N)

Sets the maximal number of iterations per call of boptimize! to N.