Exponential Family Distributions

Exponential Family

All the distributions provided by this package are members of the exponential family of distribution, i.e. they have the follotwing canonical form:

\[p(x) = \exp \{ \eta^\top T(x) - A(\eta) + B(x) \}\]

where:

  • $\eta$ is the natural parameter (scalar or vector)
  • $T(x)$ is the sufficient statistic (scalar or vector)
  • $A(\eta)$ is the log-normalizer (scalar)
  • $B(x)$ is the base measure (scalar)

Practically, the package provide the following abstract type

Which represents the supter-type of members of the exponential family.

Parameterization

All subtypes of Distribution have the following form:

struct MyDistribution{P<:AbstractParam} <: Distribution
    param::P
end

This particular form allows each distribution to be agnostic to their concrete parameterization. The parameter type inherits from:

and supports the following methods

ExpFamilyDistributions.realform — Function
realform(param)

Returns the vector of parameters as stored in param. Note that this function is just an accessor of the internal storage of the parameter, modifying the returned value should modify the parameter accordingly.

See also: naturalform.

Distribution interface

Each subtype of [Distribution] implements the following interface:

ExpFamilyDistributions.kldiv — Function
kldiv(q::T, p::T[, μ = gradlognorm(q)]) where T<:Distribution

Compute the KL-divergence between two distributions of the same type (i.e. kldiv(Normal, Normal), kldiv(Dirichlet, Dirichlet), ...). You can specify directly the expectation of the sufficient statistics μ.

ExpFamilyDistributions.splitgrad — Function
splitgrad(p, μ)

Split the gradient of the log-normalizer into its "standard" components. For instance, for the Normal distribution, the output will be the expected value of $x$ and $xxᵀ$.

ExpFamilyDistributions.stdparam — Function
stdparam(p, η)

Returns the standard parameters corresponding to the natural parameters η for distributions with the same type of p.

Utilities

The package also provides the following utility functions: