Probability Distributions

This post is more a memory-refresher list of the most famous probability distributions rather than a comprehensive explanation of anything in particular.

Discrete

Bernoulli

Toss of a coin with probability of head of $\lambda \in [0,1]$.

$$\begin{array}{}\text{(1)}& X\in \{0,1\}\end{array}$$

$$\begin{array}{}\text{(2)}& p(X=x\mid \lambda )=\{\begin{array}{ll}\lambda & \text{if}x=1\\ 1-\lambda & \text{if}x=0\end{array}\end{array}$$

Moments Example: By means of example, its moments would be computed as:

$$E[X]=\sum _{x}p(x)x=\lambda \cdot 1+(1-\lambda )\cdot 0=\lambda$$

$V[X]=E[(X-E(X){)}^2]=E[(X-\lambda {)}^2]=p(0)(0-\lambda {)}^2+p(1)(1-\lambda {)}^2=\lambda (1-\lambda )$

Categorical

Toss of a dice with $k$ faces and probabilities ${\lambda }_1,...,{\lambda }_k$. Where $\sum _i{\lambda }_i=1$, and ${\lambda }_i\in [0,1]\mathrm{\forall }i$.

$$\begin{array}{}\text{(3)}& X\in \{1:k\}\end{array}$$

$$\begin{array}{}\text{(4)}& p(X=x\mid {\lambda }_{i=1:N})=\{\begin{array}{ll}{\lambda }_1& \text{if}x=1\\ {\lambda }_2& \text{if}x=2\\ \dots \end{array}\end{array}$$

Remember: Categorical Cross-Entropy is often used as a loss function for ML supervised classification tasks, which is:

$$\mathcal{H}(P,Q)=E_P[I_Q]=-\sum _{i}p(P=p_i)\log (Q=p_i)=-\sum {\lambda }_i\log ({\hat{\lambda }}_i)$$

In classification tasks, we see our model prediction as a categorical distribution: for a given input, a certain probability is assigned to each class. In this context:

• $P$ is the “real” categorical distribution of the data. Usually the labels come as a 1-hot encoding: so the “real” distribution we want to match has probability $1$ in the correct label and $0$ in the others. ${\hat{\lambda }}_i$ indicates whether class $i$ is the correct one for a given input.

• $Q$ on the other hand, is the guess done by our model, with each ${\lambda }_i$ being the probability of classifying the input into class $i$.

Thus, the loss associated with a datapoint becomes: $-\log ({\hat{\lambda }}_i)$ (negative log of the guessed prob).

Binomial

Number of successes in $n$ Bernoullies (with mean $\lambda \in [0,1]$).

$$\begin{array}{}\text{(5)}& X\in \{1:n\}\end{array}$$

$$\begin{array}{}\text{(6)}& p(X=x\mid \lambda )=(\genfrac{}{}{0ex}{}{N}{x}){\lambda }^x(1-\lambda {)}^{n-x}\end{array}$$

Multinomial

Number of counts of a k-sided die rolled n times. I.e. $x_i$ counts the number of times side $i$ appeared when rolling the dice $n$ times, thus:

$$\begin{array}{}\text{(7)}& x_{i\in 1:k}\in \{0:n\}\end{array}$$

Where: $\sum _{i=1}^k{x}_i=n$

$$\begin{array}{}\text{(8)}& p(X=\{x_1,\dots ,x_k\}\mid N,{\lambda }_{i=1:N})=(\genfrac{}{}{0ex}{}{N}{x_1,\dots ,x_k}){\lambda }_1^{x_1}\dots {\lambda }_k^{x_k}\end{array}$$

Geometric

Counts the number of failures before the first success in a Bernoulli r.v.

$$\begin{array}{}\text{(9)}& x\in \mathbb{N}\end{array}$$

$$\begin{array}{}\text{(10)}& p(X=x\mid \lambda )=(1-\lambda {)}^{x-1}\lambda \end{array}$$

Usage example: This distribution is often used to model life expectancy of something that has a probability $\lambda$ of dying at every time-step. For instance, in RL: if life expectancy of an agent is $E[X]=\frac{1}{\lambda }$, we discount each expected future reward by a factor of $\lambda$ to account for the probability of being alive at that point (there are other reasons to discount it such as making infinite episode reward sums finite or account for the fact that most actions do not have long-lasting repercussions).

Poisson

Counts the number of random independent events happening in a fixed period of time (or space):

Imagine that on average $\lambda \in {\mathbb{R}}_{>0}$ events happen within a time period (aka Poison process). We can then get the probability of $x$ events happening in this time-period using the Poisson distribution:

$$\begin{array}{}\text{(11)}& x\in \mathbb{N}\end{array}$$

$$\begin{array}{}\text{(12)}& p(X=x\mid \lambda )=\frac{{\lambda }^x{e}^{-\lambda }}{x!}\end{array}$$

Continuous

Uniform

Assigns same likelihood to all values between a range.

$$\begin{array}{}\text{(13)}& x\in [a,b]\end{array}$$

$$\begin{array}{}\text{(14)}& PDF(x,a,b)=\frac{1}{b-a}\end{array}$$

• Is the distribution which gives the maximum entropy for $x\in [a,b]$

MLE Example: By means of example, lets see how we would compute the MLE of its parameters given a dataset $$\begin{gathered} \mathcal{D}= \\ {x_i \\ }_{i=1:n} \end{gathered}$$. If we express the likelihood of the dataset we get that:

$$\begin{array}{}\text{(15)}& \mathcal{L}\mathcal{(}\mathcal{D}\mathcal{,}\mathcal{a}\mathcal{,}\mathcal{b}\mathcal{)}={\left(\frac{1}{b-a}\right)}^n\end{array}$$

If we want to maximize the likelihood, we need to minimize $b-a$ but $x_i\in [a,b]\mathrm{\forall }i$. Thus, the minimum is achieved when:

$$\begin{array}{}\text{(16)}& a=\underset{i}{min}{x}_{i}b=\underset{i}{max}{x}_i\end{array}$$

Gaussian/Normal

$$\begin{array}{}\text{(17)}& x\in \mathbb{R}\end{array}$$

$$\begin{array}{}\text{(18)}& PDF(x,\mu ,\sigma )=\frac{1}{\sqrt{2\pi {\sigma }^2}}{e}^{-\frac{1}{2}\frac{(x-\mu {)}^2}{{\sigma }^2}}\end{array}$$

• Is the distribution which gives the maximum entropy for a fixed $\mu ,{\sigma }^2$ for $x\in \mathbb{R}$.
• A property which makes the Gaussian distribution extremely analytically tractable is its closure under linear combinations: The linear combination of Gaussian r.v.’s is Gaussian.

Weak Law of Large Numbers

The mean of a sequence of i.i.d random variables converges in probability to the expected value of the random variable as the length of that sequence tends to infinity:

$$\begin{array}{}\text{(19)}& {\overline{X}}_n\stackrel{p}{\to }E[X]\end{array}$$

This is very nice, but it only considers the mean of a SINGLE set of samples of a random variable. But what distribution does this mean follow?

Central limit theorem

The distribution of sample means of any distribution converges in distribution to a normal distribution.

$$\begin{array}{}\text{(20)}& \frac{{\overline{X}}_n-\mu }{\sigma \sqrt{n}}\stackrel{d}{\to }N(0,1),\end{array}$$

Combining these two theorems, we can assume that the mean of samples of any distribution will follow a normal distribution around the expected value if the sample size is big enough (usually 30 samples is considered big enough).

Chi-squared ${\chi }^2$

Models the sum of k squared standardized normals.

$$\begin{array}{}\text{(21)}& x\in (0,\mathrm{\infty })\end{array}$$

$$\begin{array}{}\text{(22)}& PDF(x,k)=\frac{1}{\mathrm{\Gamma }(k)2^{\frac{k}{2}}}{x}^{\frac{k}{2}-1}{e}^{-\frac{x}{2}}\end{array}$$

Exponential

Represents the distribution probability of the amount of time between two Poison-type events. Measures the amount of time probability between two Poisson-type events. $\lambda$ again is the expected number of events within the time period.

$$\begin{array}{}\text{(23)}& x\in [0,\mathrm{\infty })\end{array}$$

$$\begin{array}{}\text{(24)}& PDF(x,\lambda )=\lambda e^{-\lambda x}\end{array}$$

It is often said that “it doesn’t have memory, this happens because the occurrence of events are independent from each other. The way I picture it is with this process:

1. Throw $\lambda$ darts into a 1-dimensional target of fixed length
2. Walk through the target from side to side.

The probability distribution of time to the next dart is exponential and it doesn’t matter that you just saw one, the probability of seeing another one is completely unrelated:

$$\begin{array}{}\text{(25)}& p(x>s+t\mid x>s)=p(x>t)\end{array}$$

Gamma

The same way the exponential distribution predicts the amount of time until the first Poisson event, the Gamma distribution predicts the time until the k-th Poisson of event having rate $\lambda \frac{\text{events}}{\text{timeperiod}}$.

$$\begin{array}{}\text{(26)}& x\in (0,\mathrm{\infty })\end{array}$$

Presents two representations. One with shape parameter $k$ (“number” of events) and scale parameters $\theta =\frac{1}{\lambda }$ (inverse of Poisson rate $\lambda$):

$$\begin{array}{}\text{(27)}& PDF(x,k,\theta )=\frac{1}{\mathrm{\Gamma }(\alpha ){\theta }^k}{x}^{k-1}{e}^{-\frac{x}{\theta }}\end{array}$$

And one with shape ($\alpha =k$ “number” of events) and rate parameters $\beta =\frac{1}{\theta }$. Notice that this rate is the same as the dictated by the Poisson distribution: $\beta =\lambda$)

$$\begin{array}{}\text{(28)}& PDF(x,\alpha ,\beta )=\frac{{\beta }^{\alpha }}{\mathrm{\Gamma }(\alpha )}{x}^{\alpha -1}{e}^{-\beta x}\end{array}$$

• It is the distribution which gives a maximum entropy for a fixed $E[X]=k\theta =\frac{\alpha }{\beta }\ge 0$ for $x\in (0,\mathrm{\infty })$
• Exponential (k=1), ${\chi }^2$, and Erlang distributions are particular cases of Gamma distribution.
• It is often used as a conjugate prior of other distributions.