Divergence and Distance

 cross_entropy(P::CountVector, Q::CountVector, ::Type{T}) where {T<:MaximumLikelihood}

\[H(P,Q) = - \sum_x(P(x) \log(Q(x)))\]

Compute the cross entropy of $P$ and $Q$, given an estimator of type $T$. $P$ and $Q$ must be the same length. Both vectors are normalised. The cross entropy of a probability distribution, $P$ with itself is equal to its entropy, ie $H(P, P) = H(P)$.

Example

julia> P = cvector([1,2,3,4,3,2])
julia> Q = cvector([2,5,5,4,3,4])

julia> ce = cross_entropy(P, Q, MaximumLikelihood)
1.778564897565542

Note: not every estimator is currently supported.

kl_divergence(p::AbstractVector, q::AbstractVector)::Float64

\[D_{KL}(P ‖ Q) = \sum_{x \in X} P(x) \log \left( \frac{P(x)}{Q(x)} \right)\]

Compute the Kullback-Lebler Divergence between two discrete distributions. Both distributions needs to be defined over the same space, so length(p) == length(q). If the distributions are not normalised, they will be.

jensen_shannon_divergence(countsP::AbstractVector, countsQ::AbstractVector)
jensen_shannon_divergence(countsP::AbstractVector, countsQ::AbstractVector, estimator::Type{T}) where {T<:NonParamterisedEstimator}
jensen_shannon_divergence(countsP::AbstractVector, countsQ::AbstractVector, estimator::Type{Bayes}, α)

Compute the Jensen Shannon Divergence between discrete distributions $P$ and $q$, as represented by their histograms. If no estimator is specified, it defaults to MaximumLikelihood.

\[\widehat{JS}(p, q) = \hat{H}\left(\frac{p + q}{2} \right) - \left( \frac{H(p) + H(q)}{2} \right) \]

jensen_shannon_distance(P::AbstractVector, Q::AbstractVector, estimator)

Compute the Jensen Shannon Distance

jeffreys_divergence(p, q)
(link)[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7516653/]

\[J(p, q) = D_{KL}(p \Vert q) + D_{KL}(q \Vert p)\]