Kullback-Leibler divergence

Let $\mathbf{P}$ and $\mathbf{Q}$ be discrete probability distributions with pmfs $p$ and $q$ respectively. Let’s also assume and have a common sample space $E$. Then the KL divergence between discrete probability distributions and is defined by
$\text {KL}(\mathbf{P}, \mathbf{Q}) = \sum _{x \in E} p(x) \ln \left( \frac{p(x)}{q(x)} \right).$

• $\text {KL}(\mathbf{P}, \mathbf{Q}) \geq 0$ (nonnegative)
• $\text {KL}(\mathbf{P}, \mathbf{Q}) = 0$ only if $\mathbf{P}$ and $\mathbf{Q}$ are the same distribution (definite).