MT Page

Transformations of Probability Distributions

—

by

Mars Tonry
in ,

I will share an important formula which is skipped over in almost all calculus classes. It is the transformation of variables formula for a probability density function (pdf). I will derive the formula, and use it to transform a few probability distributions.

Let’s call this distribution X, not to be confused with the variable x (lowercase). Assume that the highlighted blue section of X is an infinitely thin rectangle, with height y and width dx.

Now, if we want to transform X into g(X), we will scale the x-axis by the function g(x). Therefore, the width of the rectangle dx will scale to become dg(x), but the area must stay constant. Let’s call the height of the transformed rectangle z.

Solving for z,

ydx=zdg(x)y\text{d}x=z\text{d}g(x)
ydxdg(x)=zy\frac{\text{d}x}{\text{d}g(x)}=z

Let’s call y “fX(x)”, and z “fg(X)(x)”, because they are the pdf for distributions X and g(X). Because we rescaled the x-axis using a function g(x), we must substitute the inverse function into the pdf to account for this.

y=fX(g−1(x))y=f_X(g^{-1}(x))

The differential term dx/dg(x) is the same as the derivative of the inverse of g(x). We must ensure that the density function is always positive by taking the absolute value of the derivative. Lastly, we must apply g(x) to the bounds. We can combine all of these facts to create the formula.

fg(X)(x)=fX(g−1(x))|ddxg−1(x)|, x∈[g(b0),g(b1)]f_{g(X)}(x)=f_X(g^{-1}(x))\left|\frac{\text{d}}{\text{d}x}g^{-1}(x)\right|,\ x\in [g(b_0),g(b_1)]

Note that if g(x) is not one-to-one, the pdf must be assembled in multiple branches.

Let’s use this formula on some common probability distributions:

Transformations on the Uniform Distribution

Let’s define U to be the uniform distribution:

U∼fU(x)=1, 0≤x≤1\text{U}\sim f_U(x)=1,\ 0\le x\le 1

To begin with, let’s add 4 to the distribution. As expected, this will shift the distribution two to the right. The inverse function of x+4 is x-4, and the only part of the distribution which is changes is the bounds.

fU+4=1|ddx(x−4)|, 4≤x≤5f_{U+4}=1\left|\frac{\text{d}}{\text{d}x}(x-4)\right|,\ 4\le x \le 5
=1, 4≤x≤5=1,\ 4\le x \le 5

Next, let’s double the output of the distribution. The inverse function of 2x is (1/2)x, so let’s apply the formula:

f2U=1|ddx12x|, 0≤x≤2f_{2U}=1\left|\frac{\text{d}}{\text{d}x}\frac{1}{2}x\right|,\ 0\le x \le 2
=12, 0≤x≤2=\frac{1}{2},\ 0\le x \le 2

Let’s do something more complicated. We will take the reciprocal of the distribution. The inverse of a reciprocal is another reciprocal, so we will substitute 1/x in the function.

f1/U=1|ddx1x|, 1≤x<∞f_{1/U}=1\left|\frac{\text{d}}{\text{d}x}\frac{1}{x}\right|,\ 1\le x < \infin
=|−1x2|,1≤x<∞=\left|-\frac{1}{x^2}\right|, 1\le x < \infin
=1x2,1≤x<∞=\frac{1}{x^2}, 1\le x < \infin

This is the standard Pareto distribution. The tail end of the distribution drops off very slowly. In fact, only 83.6% of the distribution is contained in this screenshot. This is because the uniform distribution contains many values close to zero, which blow up to infinity when taking the reciprocal.

Finally, let’s square the distribution. The inverse of x2 is the square root of x.

fU2=1|ddxx|, 0≤x≤1f_{U^2}=1\left|\frac{\text{d}}{\text{d}x}\sqrt{x}\right|,\ 0\le x\le 1
=12x, 0≤x≤1=\frac{1}{2\sqrt{x}},\ 0\le x\le 1

This function shoots up to infinity at 0. This is again due to the abundance of values near zero in the uniform distribution. Taking the square pushes even more values near zero, creating an infinite density there.

Transformations on the Normal Distribution

Let’s define N to be the standard normal distribution.

N∼fN(x)=12πe−12x2, −∞<x<∞\text{N}\sim f_N(x)=\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}x^2},\ -\infin<x<\infin

This is the most common distribution in statistics, because of the Central Limit Theorem. The Central Limit Theorem states that the sum of a large number of i.i.d (independent and identically distributed) random variables approaches the normal distribution. However, if we took the exponent of the normal distribution, we would see another form of the Central Limit Theorem: The product of i.i.d distributions approaching this new distribution.

feN(x)=12πe−12(ln x)2|ddxln x|, 0<x<∞f_{e^N}(x)=\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}\left(\text{ln}\ x\right)^2}\left|\frac{\text{d}}{\text{d}x}\text{ln}\ x\right|,\ 0<x<\infin
=12πe−12ln2x|1x|, 0<x<∞=\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}\text{ln}^2x}\left|\frac{1}{x}\right|,\ 0<x<\infin
=12πxe−12ln2x, 0<x<∞=\frac{1}{\sqrt{2\pi}x}e^{-\frac{1}{2}\text{ln}^2x},\ 0<x<\infin

The symmetric bell curve has been dramatically skewed to the right because of the exponent. Instead of equal tails, it has a sharp left tail and a fatter right tail, although this distribution isn’t necessarily heavy-tailed. This is counterintuitively called the lognormal distribution, despite being the exponent of the normal.

Lastly, let’s square the normal distribution. f(x)=x2 is not one-to-one, so we must construct both branches of its inverse.

fN2(x)=12πe−12(x)2|ddxx|+12πe−12(−x)2|ddx−x|, 0≤x<∞f_{N^2}(x)=\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}\left(\sqrt{x}\right)^2}\left|\frac{\text{d}}{\text{d}x}\sqrt{x}\right|+\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}\left(-\sqrt{x}\right)^2}\left|\frac{\text{d}}{\text{d}x}-\sqrt{x}\right|,\ 0\le x<\infin
=12πe−12x|12x|+12πe−12x|−12x|, 0≤x<∞=\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}x}\left|\frac{1}{2\sqrt{x}}\right|+\frac{1}{\sqrt{2\pi}}e^{-\frac{1}{2}x}\left|-\frac{1}{2\sqrt{x}}\right|,\ 0\le x<\infin
=12π1xe−12x, 0≤x<∞=\frac{1}{\sqrt{2\pi}}\frac{1}{\sqrt{x}}e^{-\frac{1}{2}x},\ 0\le x<\infin

This distribution has both a singularity at 0 and an infinite tail. It is even more skewed to the right than the lognormal distribution. It is a chi-squared distribution with one degree of freedom, and it shows up in statistics when dealing with more than one normally distributed quantity at once.

Comments

Leave a Reply