# Divergence and Distance

DiscreteEntropy.cross_entropyFunction
 cross_entropy(P::CountVector, Q::CountVector, ::Type{T}) where {T<:MaximumLikelihood}
$$$H(P,Q) = - \sum_x(P(x) \log(Q(x)))$$$

Compute the cross entropy of $P$ and $Q$, given an estimator of type $T$. $P$ and $Q$ must be the same length. Both vectors are normalised. The cross entropy of a probability distribution, $P$ with itself is equal to its entropy, ie $H(P, P) = H(P)$.

Example


julia> P = cvector([1,2,3,4,3,2])
julia> Q = cvector([2,5,5,4,3,4])

julia> ce = cross_entropy(P, Q, MaximumLikelihood)
1.778564897565542

Note: not every estimator is currently supported.

DiscreteEntropy.kl_divergenceFunction
kl_divergence(p::AbstractVector, q::AbstractVector)::Float64
$$$D_{KL}(P ‖ Q) = \sum_{x \in X} P(x) \log \left( \frac{P(x)}{Q(x)} \right)$$$

Compute the Kullback-Lebler Divergence between two discrete distributions. Both distributions needs to be defined over the same space, so length(p) == length(q). If the distributions are not normalised, they will be.

DiscreteEntropy.jensen_shannon_divergenceFunction
jensen_shannon_divergence(countsP::AbstractVector, countsQ::AbstractVector)
jensen_shannon_divergence(countsP::AbstractVector, countsQ::AbstractVector, estimator::Type{T}) where {T<:NonParamterisedEstimator}
jensen_shannon_divergence(countsP::AbstractVector, countsQ::AbstractVector, estimator::Type{Bayes}, α)

Compute the Jensen Shannon Divergence between discrete distributions $P$ and $q$, as represented by their histograms. If no estimator is specified, it defaults to MaximumLikelihood.

$$$\widehat{JS}(p, q) = \hat{H}\left(\frac{p + q}{2} \right) - \left( \frac{H(p) + H(q)}{2} \right)$$$
DiscreteEntropy.jeffreys_divergenceFunction
jeffreys_divergence(p, q)
(link)[https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7516653/]
$$$J(p, q) = D_{KL}(p \Vert q) + D_{KL}(q \Vert p)$$$