`BayesianExperiments.BetaStatistics`

— Type`BetaStatistics(numsuccesses, numtrials)`

Sample statistics of data needed to update a Beta distribution.

`BayesianExperiments.ChainedModel`

— Type`ChainedModel <: ProbabilisticModel`

`ChainedModel`

is a combination of `ConjugateModel`

s chained by the specified operator. It can be used to model a multiple step process.

`BayesianExperiments.ConjugateBernoulli`

— Type`ConjugateBernoulli <: ConjugateModel`

Bernoulli likelihood with Beta distribution as the conjugate prior.

```
ConjugateBernoulli(α, β) # construct a ConjugateBernoulli
update!(model, stats) # update model with statistics from data
samplepost(model, numsamples) # sampling from the posterior distribution
samplestats(model, numsamples) # sampling statistics from the data generating distribution
```

`BayesianExperiments.ConjugateExponential`

— Type`ConjugateExponential <: ConjugateModel`

Exponential likelihood with Gamma distribution as the conjugate prior.

```
ConjugateExponential(α, β) # construct a ConjugateExponential
update!(model, stats) # update model with statistics from data
samplepost(model, numsamples) # sampling from the posterior distribution
samplestats(model, numsamples) # sampling statistics from the data generating distribution
```

`BayesianExperiments.ConjugateLogNormal`

— Type`ConjugateLogNormal(μ, v, α, θ)`

A model with Normal likelihood and Normal Inverse distribution with log transformed data. Notice `LogNormal`

in `Distributions.jl`

takes mean and standard deviation of $\log(x)$ instead of $x$ as the input parameters.

```
ConjugateLogNormal(μ, v, α, θ) # construct a ConjugateLogNormal
lognormalparams(μ_logx, σ²_logx) # convert normal parameters to log-normal parameters
update!(model, stats) # update model with statistics from data
samplepost(model, numsamples) # sampling from the posterior distribution
samplestats(model, numsamples) # sampling statistics from the data generating distributio
```

`BayesianExperiments.ConjugateModel`

— Type`ConjugateModel <: ProbabilisticModel`

`ConjugateModel`

is a `ProbabilisticModel`

with a conjugate prior of the corresponding likelihood function.

`BayesianExperiments.ConjugateNormal`

— Type`ConjugateNormal <: ConjugateModel`

Normal likelihood and Normal Inverse Gamma distribution as the conjugate prior.

**Parameters**

- μ: mean of normal distribution
- v: scale variance of Normal
- α: shape of Gamma distribution
- θ: scale of Gamma distribution

```
ConjugateNormal(μ, v, α, θ) # construct a ConjugateNormal
update!(model, stats) # update model with statistics from data
samplepost(model, numsamples) # sampling from the posterior distribution
samplestats(model, numsamples) # sampling statistics from the data generating distribution
```

**References**

- The update rule for Normal distribution is based on this lecture notes.

`BayesianExperiments.ExpectedLossThresh`

— Type`ExpectedLossThresh <: StoppingRule`

The experiment has a winning model if the model has the smallest posterior expected loss, and its expected loss value is below the threshold.

**References**

`BayesianExperiments.Experiment`

— Type`Experiment`

An experiment has the models to compare, a stopping rule to make decision.

`BayesianExperiments.ExperimentABN`

— Type`ExperimentABN{T,n} <: Experiment`

An experiment with stopping rule of type `T`

and `n`

models. Models must have the same `ProbabilisticModel`

type.

`BayesianExperiments.ExperimentBF`

— Type`ExperimentBF{M} <: Experiment`

`ExperimentBF`

is an experiment using a Bayes Factor between the null and alternative hypothesis as the stopping rule.

**Constructors**

`ExperimentBF(kwargs...)`

**Keywords**

`model::M`

: Prior of effect size of alternative hypothesis`p0::Float64`

: Probablity of null hypothesis`winner::Union{String, Nothing`

: Decision to reject the null hypothesis or not`rule::BayesFactorThresh`

: Stopping rule using Bayes Factor as the threshold`stats`

: Statistics for calculating the bayes factor. Default is`nothing`

.`modelnames`

: Names of the hypotheses. Default is`["null", "alternative"]`

.

`BayesianExperiments.GammaStatistics`

— Type`GammaStatistics <: ModelStatistics`

Sample statistics of data needed to update a Gamma distribution.

`BayesianExperiments.LogNormalStatistics`

— Type`LogNormalStatistics <: ModelStatistics`

Sample statistics of data related to Log-Normal distribution.

`BayesianExperiments.NormalEffectSize`

— TypeNormalEffectSize <: EffectSizeModel

A standard effect size model has two hypotheses: $H_0$(null) an $H_1$(alternative):

- $H_0$: $\mu = m_0$
- $H_1$: $\mu ≠ m_0$

with the population mean $\mu$ and pre-specified standard deviation $\sigma$. We want to test whether $\mu$ is equal to $\mu_0$ or not.

The prior of the standard effect size is

$\delta | H_1 \sim \text{Normal}(0, \sigma_0^2)$

where $\delta$ is the standard effect size. When $\sigma_0 = 1$the prior is called *unit-information* prior.

The standard effect size $\delta$ is defined as

$\delta = \frac{\mu - \mu_0}{\sigma}.$

In practice, the standard deviations are unknown but in large sample scenario we assume they are known and use their estimates.

**Fileds**

`μ0`

: mean of null hypothesis.`σ0`

: The prior standard deviation of the effect size.`p0`

: prior belief of $H_0$. This is used to calculate the prior odds. Default $0.5$.

**Methods**

```
bayesfactor(model, stats) # calculate Bayes factor from one group statistics
bayesfactor(model, twostats) # calculate Bayes factor from two group's statistics
```

**References**

- Chapter 5 hypothesis Testing with Normal Populations in
*An Introduction to Bayesian Thinking*. - Deng, Alex, Jiannan Lu, and Shouyuan Chen. "Continuous monitoring of A/B tests without pain: Optional stopping in Bayesian testing." 2016 IEEE international conference on data science and advanced analytics (DSAA). IEEE, 2016.

`BayesianExperiments.NormalInverseGamma`

— Type`NormalInverseGamma{T<:Real} <: ContinuousUnivariateDistribution`

Normal Inverse Gamma distribution. Adapted from the `NormalInverseGamma`

in the ConjugatePriors.jl package.

`BayesianExperiments.NormalStatistics`

— Type`NormalStatistics <: ModelStatistics`

Sample statistics of data related to Normal distribution.

**Methods**

`update!(stats_old, stats_new)`

Batch update for Normal statistics.

`merge(twostats)`

Convert statistics of two independent samples into one NormalStatistics.

Mean is the difference between group 1 and group 2, minus the difference in the null hypothesis.

The standard deviation is calculated with pooled standard deviation.

The sample size is the effective sample size.

effect(stats)

Calculate effect size for one NormalStatistics or TwoNormalStatistics.

**References**

- [Batch updates for simple statistics](

https://notmatthancock.github.io/2017/03/23/simple-batch-stat-updates.html)

`BayesianExperiments.OneSidedBFThresh`

— Type`OneSidedBFThresh <: BayesFactorThresh`

The bayes factor itself is interpretable as the comparative evidence of data under the two competing hypotheses. Higher bayes factor, as defined $\text{BF}_{10}$ favours the alternative hypothesis.

In practice, a threshold can be used to make decision in bayes factor experiment. The experiment will stop when $\text{BF}_{10}$ > threshold.

In this case, the bayes factor of alternative over null is above the threshold, we can accept the alternative hypothesis. Otherwise, we don't have enough evidence to accept alternative hypotheses.

`BayesianExperiments.PosteriorSample`

— Type`PosteriorSample`

Samples from the poseteriors distribution.

`BayesianExperiments.ProbabilisticModel`

— Type`ProbabilisticModel`

`ProbabilisticModel`

is a a model of the parameters that we are interested in. The model is defined by its prior distribution and likelihood function.

`BayesianExperiments.ProbabilityBeatAllThresh`

— Type`ProbabilityBeatAllThresh <: StoppingRule`

The experiment has a winning model if probability of that model's posterior samples is larger than the alternative models is above the threshold.

`BayesianExperiments.Simulation`

— Type`Simulation(experiment, parameters, datagendists, maxsteps, onestepsizes, minsteps)`

A simulation setup includes the experiment, data generating distributions, max number of steps and minimum number of steps.

`BayesianExperiments.StudentTEffectSize`

— Type`StudentTEffectSize <: BayesFactorModel`

A model with Bayes factor from the Student's t distributions. We have a standard effect size model has two hypotheses: $H_0$(null) an $H_1$(alternative):

- $H_0$: $\mu = m_0$
- $H_1$: $\mu ≠ m_0$

The model uses the Jeffreys-Zellener-Siow (JZS) prior. More specifically, we use a Cauchy prior on $\mu$ for $H_1$

$\mu | \sigma^2 \sim \text{Cauchy}(0, r^2 \sigma^2)$

and a Jeffrey's prior on $\sigma$:

$p(\sigma^2) \propto \frac{1}{\sigma2}$

for both $H_0$ and $H_1$.

**Fields**

- $r$: Prior standard deviation of the effect size.
- $rtol$: Numerical tolerence fo the $quadgk$ function used in the denominator calculation.
- $p0$: prior belief of $H_0$. This is used to calculate the prior odds. Default $0.5$.

**References**

- Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic bulletin & review, 16(2), 225-237.

`BayesianExperiments.StudentTStatistics`

— Type`StudentTStatistics <: ModelStatistics`

Sample statistics of data related to Student's t distribution.

`BayesianExperiments.TwoSidedBFThresh`

— Type`TwoSidedBFThresh <: BayesFactorThresh`

The bayes factor itself is interpretable as the comparative evidence of data under the two competing hypotheses. Higher bayes factor, as defined $\text{BF}_{10}$ favours the alternative hypothesis.

In practice, a threshold can be used to make decision in bayes factor experiment. The experiment will stop when

- $\text{BF}_{10}$ > threshold
- $\text{BF}_{10}$ < 1/threshold

In (1), the bayes factor of alternative over null is above the threshold, we can accept the alternative hypothesis. In (2), the bayes factor is below the inverse of the threshold, we can accept the null hypothesis. Otherwise, we don't have enough evidence to accept any of these hypotheses.

`Base.convert`

— Method`convert(::DataGeneratingDistibutions, v)`

Converting DataGeneratingDistibutions to specific vector.

`BayesianExperiments.ExperimentAB`

— Method`ExperimentAB(models, rule; modelnames=nothing) where T <: StoppingRule`

An experiment of `ExperimentABN`

with two models.

`BayesianExperiments.catbyrow`

— Method`catbyrow(arr)`

Convert an array of array to a 2-dimenional matrix, the inner vector is transposed as concatenated as rows.

`BayesianExperiments.decide!`

— Method`decide!(experiment; numsamples=10_000)`

Make decision based on an experiment result and its stopping rule.

`BayesianExperiments.expectedloss`

— Method`expectedloss(modelA, modelB; lossfunc, numsamples)`

Approximating the expected loss for choosing model A over model B.

`expectedlosses(experiment::ExperimentABN; lossfunc, numsamples)`

Approximating the expected loss for all models in the experiment. For A/B/N experiment, the expected loss for each model is the maximum of the losses by comparing that model to all the other models.

`BayesianExperiments.metrics`

— Method`metrics(experiment, parameters, numsamples)`

Returns the winner's index and key metrics of the experiment.

`BayesianExperiments.samplepost`

— Method`samplepost(model, numsamples)`

Sample from the posterior distribution of the model.

`BayesianExperiments.samplestats`

— Method`samplestats(model, dist, numsamples)`

Sample from the distribution of the data generating process, and calculate the corresponding statistics for the model.

`BayesianExperiments.toparameter`

— Method`toparameter(parameters)`

Convert a parameter vector of length 1 to a single parameter.

`BayesianExperiments.upliftloss`

— Method`upliftloss(a, b)`

The lost uplift when we select "a" but "b" is actually better. Margin is the minimum lift level of a comparing to b.