Radon-Nikodym & Probability Densities

Let $\mu$ and $\nu$ be measures on a measurable space $(\Omega, \mathcal{F})$. We say $\nu$ is absolutely continuous with respect to $\mu$, written $\nu \ll \mu$, if for every $A \in \mathcal{F}$,

$\mu(A) = 0 \;\;\Longrightarrow\;\; \nu(A) = 0.$

In words: every $\mu$-null set is also a $\nu$-null set. The relationship is one-directional — $\nu \ll \mu$ does not imply $\mu \ll \nu$.

Two measures $\mu$ and $\nu$ on $(\Omega, \mathcal{F})$ are mutually singular, written $\mu \perp \nu$, if there exist disjoint sets $A, B \in \mathcal{F}$ with $A \cup B = \Omega$ such that

$\mu(A) = 0 \quad \text{and} \quad \nu(B) = 0.$

In words: the two measures live on disjoint pieces of $\Omega$ — $\mu$ assigns all its mass to $B$, and $\nu$ assigns all its mass to $A$. They cannot see each other.

Let $P$ be the standard normal probability measure on $(\mathbb{R}, \mathcal{B})$ and let $\lambda$ denote Lebesgue measure. If $\lambda(A) = 0$ — say, $A$ is a countable set or a Cantor-type set of Lebesgue measure zero — then

$P(A) = \int_A \frac{1}{\sqrt{2\pi}} e^{-x^2/2} \, dx = 0,$

because the Lebesgue integral over a null set is always zero. So $P \ll \lambda$. The density $\frac{1}{\sqrt{2\pi}} e^{-x^2/2}$ is — as we will see in Section 3 — the Radon-Nikodym derivative $\frac{dP}{d\lambda}$. Every Gaussian distribution you have ever computed with is a worked example of absolute continuity.

Let $\delta_0$ be the Dirac measure at $0$: $\delta_0(A) = 1$ if $0 \in A$, else $0$. Then $\delta_0 \perp \lambda$. To verify, take $A = \{0\}$ and $B = \mathbb{R} \setminus \{0\}$. Then $\lambda(A) = 0$ (a single point has Lebesgue measure zero) and $\delta_0(B) = 0$ (the Dirac mass is not at any point of $B$). The two measures partition $\mathbb{R}$ into the single point where $\delta_0$ lives and the rest of the line where $\lambda$ lives. They cannot be combined into a density relationship — there is no function $f$ with $\delta_0(A) = \int_A f \, d\lambda$, because integrating any Lebesgue-integrable function over $\{0\}$ gives zero.

Let $\#$ be counting measure on $\mathbb{R}$ that assigns measure $1$ to each integer and $0$ to non-integers; equivalently, $\#(A) = |A \cap \mathbb{Z}|$. Then $\# \perp \lambda$. Take $A = \mathbb{Z}$ and $B = \mathbb{R} \setminus \mathbb{Z}$. Then $\lambda(A) = 0$ (a countable set has Lebesgue measure zero) and $\#(B) = 0$ (no integers in $B$). Discrete and continuous measures live on disjoint supports. This is why you cannot write a Poisson distribution as $dP/d\lambda$ — the Poisson measure is supported on $\{0, 1, 2, \ldots\}$, which is a $\lambda$-null set.

The reader may know “absolutely continuous function” from calculus — a function $F: [a, b] \to \mathbb{R}$ is absolutely continuous if for every $\varepsilon > 0$ there is a $\delta > 0$ such that any finite collection of disjoint subintervals with total length less than $\delta$ produces total variation less than $\varepsilon$. These two concepts are related but distinct: a function $F$ is absolutely continuous on $[a, b]$ if and only if there exists $f \in L^1([a, b])$ with $F(x) = F(a) + \int_a^x f(t) \, dt$, and equivalently the signed measure $\nu_F(A) = \int_A F'(x) \, dx$ is absolutely continuous (in our measure-theoretic sense) with respect to Lebesgue measure. The function-theoretic notion is a special case of the measure-theoretic one.

The Radon-Nikodym theorem requires both $\mu$ and $\nu$ to be $\sigma$-finite: $\Omega$ is a countable union of measurable sets of finite measure under each. Without this, the theorem can fail. The standard counterexample is counting measure on $\mathbb{R}$ (which is not $\sigma$-finite — the only set of finite counting measure is a finite set, and $\mathbb{R}$ is not a countable union of finite sets) compared with Lebesgue measure: $\lambda \ll \#$ holds vacuously (the only $\#$-null set is $\emptyset$), but no integrable density can exist because $\lambda$ and $\#$ disagree on every uncountable measurable set. In probability applications, $\sigma$-finiteness is automatic: every probability measure is finite (hence $\sigma$-finite), so this hypothesis is rarely something the practitioner needs to check.

Let $\mu$ and $\nu$ be $\sigma$-finite measures on $(\Omega, \mathcal{F})$ with $\nu \ll \mu$. If there exists a non-negative measurable function $f \in L^1(\mu)$ such that

$\nu(A) = \int_A f \, d\mu \quad \text{for every } A \in \mathcal{F},$

then $f$ is called the Radon-Nikodym derivative of $\nu$ with respect to $\mu$, written

$f = \frac{d\nu}{d\mu}.$

The notation is deliberately Leibniz-style: $f$ behaves like a derivative in several senses (chain rule, inverse rule), as we will see in Section 5.

If $f$ and $g$ are both Radon-Nikodym derivatives of $\nu$ with respect to $\mu$, then $f = g$ $\mu$-almost everywhere.

Suppose $\int_A f \, d\mu = \nu(A) = \int_A g \, d\mu$ for every measurable $A$, so

$\int_A (f - g) \, d\mu = 0 \quad \text{for every } A \in \mathcal{F}.$

Let $A_+ = \{x : f(x) > g(x)\}$. Then $f - g > 0$ on $A_+$, and $\int_{A_+} (f - g) \, d\mu = 0$ forces $\mu(A_+) = 0$ (an integral of a strictly positive function over a set is zero only when the set has measure zero). Similarly, with $A_- = \{x : g(x) > f(x)\}$, we get $\mu(A_-) = 0$. So $\mu(\{f \neq g\}) = \mu(A_+ \cup A_-) = 0$, which is exactly the statement that $f = g$ $\mu$-a.e.

Every probability density function the reader has ever computed with is a Radon-Nikodym derivative w.r.t. Lebesgue measure on $\mathbb{R}$. For the standard normal,

$\frac{dP_{\text{normal}}}{d\lambda}(x) = \frac{1}{\sqrt{2\pi}} \, e^{-x^2/2}.$

For the exponential distribution with rate $\theta > 0$,

$\frac{dP_{\text{exp}}}{d\lambda}(x) = \theta \, e^{-\theta x} \, \mathbf{1}_{x \geq 0}.$

For the Beta$(\alpha, \beta)$ distribution on $[0, 1]$,

$\frac{dP_{\text{Beta}}}{d\lambda}(x) = \frac{x^{\alpha - 1} (1 - x)^{\beta - 1}}{B(\alpha, \beta)} \, \mathbf{1}_{[0, 1]}(x),$

where $B(\alpha, \beta)$ is the Beta function. In each case the function on the right is the R-N derivative of the probability measure on the left with respect to Lebesgue measure on $\mathbb{R}$. The defining property $\nu(A) = \int_A f \, d\mu$ becomes the familiar identity $P(A) = \int_A p(x) \, dx$.

Let $\mu$ be counting measure on $\{1, 2, 3, \ldots\}$ (assigning measure $1$ to each integer) and let $\nu$ be the geometric distribution with parameter $\frac{1}{2}$, so $\nu(\{k\}) = 2^{-k}$. Then $\nu \ll \mu$ (any $\mu$-null set is empty), and the R-N derivative is

$\frac{d\nu}{d\mu}(k) = 2^{-k}.$

The defining property reduces to $\nu(A) = \sum_{k \in A} 2^{-k}$ for $A \subseteq \mathbb{Z}_{\geq 1}$. The R-N derivative of a discrete probability measure with respect to counting measure is exactly the probability mass function. The “PDF vs PMF” distinction every student learns is purely about which dominating measure you take the derivative against — Lebesgue for continuous distributions, counting for discrete ones. The Radon-Nikodym framework unifies them.

Let $(\Omega, \mathcal{F}, \mu)$ be a $\sigma$-finite measure space and let $\nu$ be a $\sigma$-finite measure on $(\Omega, \mathcal{F})$ with $\nu \ll \mu$. Then there exists a non-negative measurable function $f \in L^1(\mu)$ such that

$\nu(A) = \int_A f \, d\mu \quad \text{for every } A \in \mathcal{F}.$

The function $f$ is unique $\mu$-a.e. (Proposition 1) and is called the Radon-Nikodym derivative $\frac{d\nu}{d\mu}$.

Following von Neumann, we reduce the existence of the density to a single application of $L^2$ orthogonal projection. The argument has four stages.

Stage 1 — Reduction to finite measures. By $\sigma$-finiteness, we can write $\Omega$ as a disjoint union $\Omega = \bigsqcup_n E_n$ with $\mu(E_n) < \infty$ and $\nu(E_n) < \infty$ for every $n$. If we can produce a density $f_n$ on each $E_n$ — that is, a function with $\nu|_{E_n}(A) = \int_A f_n \, d\mu|_{E_n}$ for every $A \subseteq E_n$ — then the function $f = \sum_n f_n \cdot \mathbf{1}_{E_n}$ is a density for $\nu$ on all of $\Omega$. So it suffices to prove the theorem under the additional assumption that both $\mu$ and $\nu$ are finite. Assume from here on that $\mu(\Omega) < \infty$ and $\nu(\Omega) < \infty$.

Stage 2 — The auxiliary measure $\rho = \mu + \nu$. Define $\rho = \mu + \nu$, the sum measure. Both $\mu$ and $\nu$ are absolutely continuous with respect to $\rho$ (every $\rho$-null set is in particular a $\mu$-null set and a $\nu$-null set). Now consider the linear functional $\phi: L^2(\rho) \to \mathbb{R}$ defined by

$\phi(g) = \int g \, d\nu.$

This is linear in $g$. To check it is bounded, apply the Cauchy-Schwarz inequality in $L^2(\rho)$:

$|\phi(g)| = \left|\int g \, d\nu\right| \leq \int |g| \, d\nu \leq \int |g| \, d\rho.$

The last step uses $\nu \leq \rho$ pointwise as set functions. Now apply Cauchy-Schwarz to bound the $L^1(\rho)$ integral by an $L^2(\rho)$ integral:

$\int |g| \, d\rho = \int |g| \cdot 1 \, d\rho \leq \|g\|_{L^2(\rho)} \cdot \|1\|_{L^2(\rho)} = \rho(\Omega)^{1/2} \|g\|_{L^2(\rho)}.$

So $|\phi(g)| \leq \rho(\Omega)^{1/2} \|g\|_{L^2(\rho)}$, which is exactly the statement that $\phi$ is a bounded linear functional on $L^2(\rho)$ with operator norm at most $\rho(\Omega)^{1/2}$.

Stage 3 — Riesz representation via $L^2$ projection. By Topic 27 §9 Proposition 3 — or equivalently the Riesz representation theorem for Hilbert spaces, which we previewed in Section 9 of that topic — every bounded linear functional on a Hilbert space is represented by an inner product with a unique element of the space. Applied to our $\phi$ on the Hilbert space $L^2(\rho)$, this gives a function $h \in L^2(\rho)$ such that

$\phi(g) = \langle g, h \rangle_{L^2(\rho)} = \int g h \, d\rho \quad \text{for every } g \in L^2(\rho).$

Combining with the definition of $\phi$, we get the integral identity that drives the rest of the proof:

\int g \, d\nu = \int g h \, d\rho \quad \text{for every } g \in L^2(\rho). \tag{$\ast$}

This is the entire content of Topic 27’s projection theorem applied here. Everything in Stage 4 is bookkeeping that turns $h$ into the actual R-N derivative.

Stage 4 — Extract the derivative. Substitute $g = \mathbf{1}_A$ in $(\ast)$ for an arbitrary measurable set $A$ (this is allowed because $\rho$ is finite, so $\mathbf{1}_A \in L^2(\rho)$). The left side becomes $\nu(A)$, and the right side becomes $\int_A h \, d\rho = \int_A h \, d\mu + \int_A h \, d\nu$. So

$\nu(A) = \int_A h \, d\mu + \int_A h \, d\nu.$

Rearranging,

\int_A (1 - h) \, d\nu = \int_A h \, d\mu \quad \text{for every } A \in \mathcal{F}. \tag{$\ast\ast$}

We claim $0 \leq h \leq 1$ $\rho$-a.e. To see $h \geq 0$, choose $A = \{h < 0\}$ in $(\ast\ast)$: the right side is $\int_A h \, d\mu \leq 0$ (because $h < 0$ on $A$), and the left side is $\int_A (1 - h) \, d\nu \geq 0$ (because $1 - h > 1$ on $A$ and $\nu$ is positive). Both being equal forces both to be zero, which forces $\mu(A) = 0$ and $\nu(A) = 0$, hence $\rho(A) = 0$. So $h \geq 0$ $\rho$-a.e. The argument that $h \leq 1$ is symmetric: choose $A = \{h > 1\}$, observe that $1 - h < 0$ on $A$ while $h > 0$, and conclude both sides are zero so $\rho(A) = 0$.

Now consider the set $\{h = 1\}$. Plugging $A = \{h = 1\}$ into $(\ast\ast)$ gives $\int_{\{h=1\}} 0 \, d\nu = \int_{\{h=1\}} h \, d\mu$, hence $\mu(\{h = 1\}) = 0$ (since $h = 1 > 0$ on the set). By absolute continuity $\nu \ll \mu$, this forces $\nu(\{h = 1\}) = 0$. So we may discard the set $\{h = 1\}$ — it has both $\mu$-measure zero and $\nu$-measure zero — and assume $0 \leq h < 1$ on $\Omega$.

Define

$f = \frac{h}{1 - h} \quad \text{on } \{h < 1\}.$

Then $f \geq 0$ everywhere (since $h \in [0, 1)$). The remaining step is to upgrade the test-function identity $(\ast\ast)$ from indicators to functions large enough to invert $(1 - h)$ — this is the only place the algebra needs care.

The integral identity $(\ast\ast)$ — namely $\int_A (1 - h) \, d\nu = \int_A h \, d\mu$ for every measurable $A$ — extends from indicators to non-negative measurable test functions by the standard machine: linearity gives it for simple functions, and the Monotone Convergence Theorem (Topic 26 Theorem 1) extends it to non-negative measurables. Equivalently, the two measures $(1 - h) \, d\nu$ and $h \, d\mu$ on $\{h < 1\}$ agree on every measurable set, hence are equal as measures.

Now apply this extended identity to the test function $\varphi = \mathbf{1}_A / (1 - h)$, where $A \subseteq \{h < 1 - \tfrac{1}{n}\}$ for some $n \geq 1$ (so that $\varphi$ is bounded by $n$ and hence integrable). On this restricted set the substitution gives

$\int_A \mathbf{1}_A \, d\nu = \int_A \frac{\mathbf{1}_A}{1 - h} \cdot (1 - h) \, d\nu = \int_A \frac{\mathbf{1}_A}{1 - h} \cdot h \, d\mu = \int_A \frac{h}{1 - h} \, d\mu = \int_A f \, d\mu.$

So $\nu(A) = \int_A f \, d\mu$ for every $A \subseteq \{h < 1 - \tfrac{1}{n}\}$. Taking $n \to \infty$ and applying MCT to the increasing sequence $\{h < 1 - \tfrac{1}{n}\} \uparrow \{h < 1\}$ extends the identity to all measurable $A \subseteq \{h < 1\}$. Combined with the earlier observation that $\nu(\{h = 1\}) = 0$, we get $\nu(A) = \int_A f \, d\mu$ for every measurable $A \subseteq \Omega$.

This is exactly the Radon-Nikodym identity: $f$ is the density we sought. Finiteness of $f$ in $L^1(\mu)$ follows from $\nu(\Omega) < \infty$, since $\int_\Omega f \, d\mu = \nu(\Omega) < \infty$.

The classical proof of Radon-Nikodym (via the Hahn decomposition theorem) requires building two pages of signed-measure machinery from scratch — positive sets, negative sets, total variation, and the Hahn decomposition itself. Von Neumann’s proof reduces the entire theorem to a single application of $L^2$ projection, which we already proved in Topic 27. The argument is essentially: “a bounded linear functional on $L^2$ is representable as an inner product, and the R-N derivative falls out of the algebra.” This is one of the most elegant proof strategies in analysis. It also closes Topic 27’s promise: the $L^2$ projection theorem we previewed in §9 is the engine that drives this proof. The function-space machinery built one topic ago has paid off in full.

Let $\mu$, $\nu$, and $\lambda$ be $\sigma$-finite measures on $(\Omega, \mathcal{F})$ with $\nu \ll \mu \ll \lambda$. Then $\nu \ll \lambda$ and

$\frac{d\nu}{d\lambda} = \frac{d\nu}{d\mu} \cdot \frac{d\mu}{d\lambda} \qquad \lambda\text{-a.e.}$

That $\nu \ll \lambda$ follows immediately from the chain of implications: if $\lambda(A) = 0$ then $\mu(A) = 0$ (by $\mu \ll \lambda$), and then $\nu(A) = 0$ (by $\nu \ll \mu$). For the formula, let $f = d\nu/d\mu$ and $g = d\mu/d\lambda$, both of which exist by Theorem 1. We need to show $\nu(A) = \int_A f g \, d\lambda$ for every $A$, because then by uniqueness (Proposition 1), $fg$ must equal $d\nu/d\lambda$ a.e.

Start with the definition of $f$: $\nu(A) = \int_A f \, d\mu$. We want to rewrite the right side as a $\lambda$-integral. The key fact is that for any non-negative measurable function $\phi$,

$\int \phi \, d\mu = \int \phi \cdot g \, d\lambda$

where $g = d\mu/d\lambda$. This identity follows by the standard machine: it holds for $\phi = \mathbf{1}_B$ (just the definition of $g$ as the density), extends to simple functions by linearity, extends to non-negative measurables by the Monotone Convergence Theorem (Topic 26 Theorem 1), and extends to $L^1(\mu)$ functions by the $f = f^+ - f^-$ decomposition. Apply this with $\phi = f \cdot \mathbf{1}_A$:

$\nu(A) = \int_A f \, d\mu = \int (f \cdot \mathbf{1}_A) \, d\mu = \int (f \cdot \mathbf{1}_A) \cdot g \, d\lambda = \int_A f g \, d\lambda.$

This holds for every measurable $A$, so by Proposition 1, $fg = d\nu/d\lambda$ $\lambda$-a.e.

Let $\mu$ and $\nu$ be $\sigma$-finite measures on $(\Omega, \mathcal{F})$ with $\nu \ll \mu$ and $\mu \ll \nu$ (mutual absolute continuity). Then

$\frac{d\mu}{d\nu} = \left(\frac{d\nu}{d\mu}\right)^{-1} \quad \mu\text{-a.e. and } \nu\text{-a.e.}$

This follows from the chain rule with $\lambda = \nu$: $\frac{d\nu}{d\nu} = 1 = \frac{d\nu}{d\mu} \cdot \frac{d\mu}{d\nu}$, so the two factors are reciprocals where they are nonzero. Mutual absolute continuity guarantees that the set where one factor vanishes is a null set under both measures, so the reciprocal is well-defined a.e.

Let $X$ be a real-valued random variable with density $f_X$ with respect to Lebesgue measure $\lambda$. Let $g: \mathbb{R} \to \mathbb{R}$ be a strictly monotone differentiable function with nowhere-zero derivative — that is, a $C^1$ diffeomorphism of $\mathbb{R}$ onto its image — and let $Y = g(X)$. The pushforward measure $P_Y(A) = P_X(g^{-1}(A))$ has a density with respect to Lebesgue measure on the image. By the chain rule for R-N derivatives applied to the pushforward,

$\frac{dP_Y}{d\lambda}(y) = f_X(g^{-1}(y)) \cdot \frac{1}{|g'(g^{-1}(y))|}.$

This is the change-of-variables formula for probability densities that every introductory probability course states without proof. The factor $1 / |g'|$ is the Jacobian of the inverse transformation, and the entire formula is just the chain rule for R-N derivatives applied in the special case of a bijection between two open subsets of $\mathbb{R}$. The same formula in higher dimensions involves the Jacobian determinant of the inverse map — again as an application of the chain rule, now for measures on $\mathbb{R}^n$.

R-N derivatives satisfy a chain rule (Theorem 2), an inverse rule (Proposition 2), and — with the right definitions — a kind of product rule. This is exactly the algebra of classical derivatives, and the Leibniz-style notation $d\nu/d\mu$ was chosen to make the analogy explicit. But the analogy has a limit: the R-N derivative is an equivalence class of functions in $L^1(\mu)$ (modulo $\mu$-null sets), not a number. Identities like the chain rule hold $\lambda$-almost everywhere, not pointwise. When using R-N notation, always remember the “a.e.” qualifier — it is the price of generality, and dropping it is the most common bookkeeping error in measure-theoretic probability.

A signed measure on $(\Omega, \mathcal{F})$ is a function $\nu: \mathcal{F} \to [-\infty, \infty]$ that takes at most one of the values $\pm \infty$, satisfies $\nu(\emptyset) = 0$, and is countably additive: for every disjoint sequence $(A_n) \subseteq \mathcal{F}$,

$\nu\left(\bigsqcup_{n} A_n\right) = \sum_{n} \nu(A_n).$

Positive measures are the special case of signed measures that happen to take only non-negative values.

The total variation of a signed measure $\nu$ on $(\Omega, \mathcal{F})$ is the positive measure $|\nu|$ defined by

$|\nu|(A) = \sup \left\{ \sum_{i=1}^{n} |\nu(A_i)| : A = \bigsqcup_{i=1}^{n} A_i, \, A_i \in \mathcal{F} \right\}$

where the supremum runs over all finite measurable partitions of $A$. The total variation $|\nu|$ measures the total mass moved by $\nu$, treating positive and negative contributions equally.

Every signed measure $\nu$ with $|\nu|(\Omega) < \infty$ can be written uniquely as

$\nu = \nu^+ - \nu^-$

where $\nu^+$ and $\nu^-$ are positive (finite) measures with $\nu^+ \perp \nu^-$. Moreover, $|\nu| = \nu^+ + \nu^-$.

We sketch the proof; the full Hahn decomposition argument is in Folland §3.1. The Hahn decomposition theorem (which we state without proof here) gives disjoint sets $P, N \in \mathcal{F}$ with $\Omega = P \cup N$ such that $\nu(A) \geq 0$ for every measurable $A \subseteq P$ and $\nu(A) \leq 0$ for every measurable $A \subseteq N$. Define

$\nu^+(A) = \nu(A \cap P), \qquad \nu^-(A) = -\nu(A \cap N).$

Both are positive measures by construction: $\nu^+$ is non-negative because $A \cap P \subseteq P$, and $\nu^-$ is non-negative because $\nu \leq 0$ on subsets of $N$ (so $-\nu \geq 0$). They are mutually singular because they live on disjoint sets ($P$ and $N$). The sum recovers $\nu$ because $\nu(A) = \nu(A \cap P) + \nu(A \cap N) = \nu^+(A) - \nu^-(A)$. Uniqueness of the decomposition follows from the mutual singularity of $\nu^+$ and $\nu^-$: any other pair of positive mutually-singular measures with the same difference must coincide with these on the sets $P$ and $N$.

We keep signed-measure theory deliberately minimal because the primary applications in this topic — probability densities, conditional expectation, importance sampling — are about positive measures. Jordan decomposition appears here for two reasons. First, it gives the cleanest extension of the R-N theorem to signed measures: if $\nu$ is a signed measure with $|\nu| \ll \mu$, then $\nu^+ \ll \mu$ and $\nu^- \ll \mu$, so applying Theorem 1 to each piece gives $d\nu^+/d\mu$ and $d\nu^-/d\mu$, and we define $d\nu/d\mu = d\nu^+/d\mu - d\nu^-/d\mu$ as a function in $L^1(\mu)$. Second, the conditional-expectation construction in Section 9 applies to any $L^1$ random variable $X$, not just non-negative ones — and that argument relies on the signed-measure version of R-N to handle $X = X^+ - X^-$ properly.

Let $\mu$ and $\nu$ be $\sigma$-finite measures on $(\Omega, \mathcal{F})$. Then $\nu$ decomposes uniquely as

$\nu = \nu_{\mathrm{ac}} + \nu_{\mathrm{sing}}$

where $\nu_{\mathrm{ac}} \ll \mu$ and $\nu_{\mathrm{sing}} \perp \mu$. In particular, $\nu_{\mathrm{ac}}$ has a Radon-Nikodym derivative $d\nu_{\mathrm{ac}}/d\mu \in L^1(\mu)$, and $\nu_{\mathrm{sing}}$ is concentrated on a $\mu$-null set.

The cleanest worked example is a measure built from three pieces of mismatched type. Let

$\nu = \tfrac{1}{3} \, \lambda|_{[0, 1]} \;+\; \tfrac{1}{3} \, \delta_0 \;+\; \tfrac{1}{3} \, \mu_C$

on $\mathbb{R}$, where $\lambda|_{[0,1]}$ is Lebesgue measure restricted to $[0, 1]$, $\delta_0$ is the Dirac mass at $0$, and $\mu_C$ is the Cantor measure (the probability measure whose CDF is the Devil’s staircase from Topic 25). Decompose $\nu$ with respect to Lebesgue measure $\lambda$ on $\mathbb{R}$. The absolutely continuous part is

$\nu_{\mathrm{ac}} = \tfrac{1}{3} \, \lambda|_{[0,1]}, \quad \text{with density } \frac{d\nu_{\mathrm{ac}}}{d\lambda} = \tfrac{1}{3} \, \mathbf{1}_{[0, 1]}.$

The singular part is

$\nu_{\mathrm{sing}} = \tfrac{1}{3} \, \delta_0 + \tfrac{1}{3} \, \mu_C.$

The Dirac component sits on the single point $\{0\}$ (Lebesgue measure zero), and the Cantor measure is supported on the Cantor set (also Lebesgue measure zero), so $\nu_{\mathrm{sing}} \perp \lambda$ via the partition $A = \{0\} \cup \mathcal{C}$ versus $B = \mathbb{R} \setminus A$. The singular part itself can be split further into a discrete component ($\delta_0$, atoms) and a singular continuous component ($\mu_C$, no atoms but no density). The full Lebesgue decomposition of any $\sigma$-finite measure on $\mathbb{R}$ has at most these three pieces: absolutely continuous, discrete, and singular continuous.

The Lebesgue decomposition theorem can be read off the same von Neumann argument that proved Radon-Nikodym. Recall the function $h \in L^2(\rho)$ from Stage 3 of the proof, where $\rho = \mu + \nu$. The set $\{h = 1\}$ is exactly where the absolute-continuity argument forced $\mu(\{h = 1\}) = 0$; on this set, $\nu$ lives on a $\mu$-null set, which is the singular part. On $\{h < 1\}$, the algebra produced the density $f = h/(1 - h)$, which is the absolutely continuous part. So defining $\nu_{\mathrm{sing}}(A) = \nu(A \cap \{h = 1\})$ and $\nu_{\mathrm{ac}}(A) = \nu(A \cap \{h < 1\}) = \int_A f \, d\mu$ gives the decomposition. The full derivation is in Stein and Shakarchi, Chapter 6.

Let $P$ be a probability measure on $(\mathbb{R}^n, \mathcal{B}(\mathbb{R}^n))$ with $P \ll \lambda$, where $\lambda$ is Lebesgue measure on $\mathbb{R}^n$. The probability density function of $P$ is

$p = \frac{dP}{d\lambda},$

the Radon-Nikodym derivative of $P$ with respect to Lebesgue measure. It satisfies (i) $p \geq 0$ $\lambda$-a.e., (ii) $\int_{\mathbb{R}^n} p \, d\lambda = 1$, and (iii) $P(A) = \int_A p \, d\lambda$ for every Borel $A \subseteq \mathbb{R}^n$.

The Beta$(\alpha, \beta)$ probability measure $P$ on $[0, 1]$ has

$\frac{dP}{d\lambda}(x) = \frac{x^{\alpha - 1}(1 - x)^{\beta - 1}}{B(\alpha, \beta)} \, \mathbf{1}_{[0, 1]}(x),$

where $B(\alpha, \beta) = \int_0^1 t^{\alpha - 1}(1 - t)^{\beta - 1} \, dt$ is the Beta function. The integral $\int_0^1 \frac{dP}{d\lambda} \, d\lambda = 1$ is exactly the definition of $B(\alpha, \beta)$. The shape of the density encodes how $P$ distributes mass relative to the uniform distribution: when $\alpha = \beta = 1$, the density is constant and $P$ is uniform; when $\alpha > \beta$ the mass shifts right; when $\alpha < 1$ or $\beta < 1$ the density blows up at an endpoint. The viz in Section 3 lets the reader explore this dependence interactively.

The Cantor measure $\mu_C$ on $[0, 1]$ is not absolutely continuous with respect to Lebesgue measure: $\mu_C$ is supported on the Cantor set $\mathcal{C}$, which has Lebesgue measure zero. So $\mu_C \perp \lambda$, and $d\mu_C/d\lambda$ does not exist. Not every probability measure has a density — only the absolutely continuous ones. The Cantor measure is the canonical example of a probability measure on $[0, 1]$ with no density: it is continuous (no atoms — every singleton has measure zero) but not absolutely continuous (it lives on a Lebesgue-null set). It is the singular continuous part of the Lebesgue decomposition. Discrete distributions (like the Poisson or geometric) also have no density with respect to Lebesgue, but they do have densities with respect to counting measure — see Example 5.

The phrase “the density of the Poisson distribution is $e^{-\theta} \theta^k / k!$” is shorthand for $dP/d\#(k) = e^{-\theta} \theta^k / k!$, where $\#$ is counting measure on $\{0, 1, 2, \ldots\}$. The same Poisson measure has no density with respect to Lebesgue measure on $\mathbb{R}$ — it is supported on the integers, which form a Lebesgue-null set. Always ask: “density with respect to what?” The answer is almost always “Lebesgue measure” for continuous distributions and “counting measure” for discrete ones, but the framework allows densities with respect to any dominating measure, and exotic choices (like the Cantor measure on $[0, 1]$) are sometimes useful in advanced applications.

Let $(\Omega, \mathcal{F}, P)$ be a probability space, $X \in L^1(P)$ an integrable random variable, and $\mathcal{G} \subseteq \mathcal{F}$ a sub-$\sigma$-algebra. The conditional expectation of $X$ given $\mathcal{G}$ is the (a.e.-unique) $\mathcal{G}$-measurable function $E[X \mid \mathcal{G}] \in L^1(P)$ satisfying

$\int_A E[X \mid \mathcal{G}] \, dP = \int_A X \, dP \quad \text{for every } A \in \mathcal{G}.$

In words: $E[X \mid \mathcal{G}]$ has the same integral as $X$ over every set you can describe using only the information in $\mathcal{G}$.

Under the conditions of Definition 7, $E[X \mid \mathcal{G}]$ exists and is unique $P$-almost everywhere.

Define the signed measure $\nu$ on the sub-$\sigma$-algebra $(\Omega, \mathcal{G})$ by

$\nu(A) = \int_A X \, dP \quad \text{for } A \in \mathcal{G}.$

Countable additivity of $\nu$ follows from the Dominated Convergence Theorem (Topic 26 Theorem 3). Then $\nu \ll P|_\mathcal{G}$: if $A \in \mathcal{G}$ and $P(A) = 0$, then $\nu(A) = \int_A X \, dP = 0$ because the Lebesgue integral over a null set is zero. Both measures are $\sigma$-finite on $\mathcal{G}$ (since $P$ is finite). By the Radon-Nikodym theorem extended to signed measures via Jordan decomposition (Section 6), there exists a $\mathcal{G}$-measurable function $f \in L^1(P|_\mathcal{G})$ with

$\nu(A) = \int_A f \, dP \quad \text{for every } A \in \mathcal{G}.$

This $f$ is $\mathcal{G}$-measurable by the way the R-N theorem produces densities (the derivative is constructed inside the function space tied to the $\sigma$-algebra), and it satisfies the defining property of conditional expectation. Set $E[X \mid \mathcal{G}] = f$. Uniqueness $P$-a.e. follows from Proposition 1 applied to $P|_\mathcal{G}$.

Let $X, Y \in L^1(P)$ and let $\mathcal{H} \subseteq \mathcal{G} \subseteq \mathcal{F}$ be sub-$\sigma$-algebras of $\mathcal{F}$.

(i) Linearity. $E[aX + bY \mid \mathcal{G}] = a E[X \mid \mathcal{G}] + b E[Y \mid \mathcal{G}]$ a.e., for any constants $a, b \in \mathbb{R}$.

(ii) Tower property. $E[E[X \mid \mathcal{G}] \mid \mathcal{H}] = E[X \mid \mathcal{H}]$ a.e. Conditioning twice on a coarser-and-finer pair collapses to conditioning once on the coarser one.

(iii) Conditional Jensen. If $\phi: \mathbb{R} \to \mathbb{R}$ is convex and $\phi(X) \in L^1(P)$, then $\phi(E[X \mid \mathcal{G}]) \leq E[\phi(X) \mid \mathcal{G}]$ a.e.

(iv) Independence. If $X$ is independent of $\mathcal{G}$ — meaning the $\sigma$-algebra $\sigma(X)$ generated by $X$ is independent of $\mathcal{G}$ — then $E[X \mid \mathcal{G}] = E[X]$ a.e. (the constant function equal to the unconditional mean).

All four properties follow directly from the defining integral identity by routine measure-theoretic arguments — see Billingsley §34.

Let $X \in L^2(\Omega, \mathcal{F}, P)$ and let $\mathcal{G} \subseteq \mathcal{F}$ be a sub-$\sigma$-algebra. Then $E[X \mid \mathcal{G}]$ is the unique element of $L^2(\Omega, \mathcal{G}, P)$ that minimizes

$\|X - Y\|_{L^2(P)}^2 = \int (X - Y)^2 \, dP$

over all $\mathcal{G}$-measurable $Y \in L^2(P)$. Equivalently, $E[X \mid \mathcal{G}]$ is the orthogonal projection of $X$ onto the closed subspace $L^2(\Omega, \mathcal{G}, P) \subseteq L^2(\Omega, \mathcal{F}, P)$.

Let $V = L^2(\Omega, \mathcal{G}, P)$ and $H = L^2(\Omega, \mathcal{F}, P)$. We claim $V$ is a closed subspace of $H$. It is clearly a linear subspace. For closedness, suppose $(Y_n) \subseteq V$ converges to $Y$ in $H$ — that is, $\|Y_n - Y\|_{L^2(P)} \to 0$. By the Riesz-Fischer theorem from Topic 27, $L^2$ convergence implies a subsequence converges $P$-a.e. Pass to that subsequence. Since each $Y_n$ is $\mathcal{G}$-measurable and the pointwise a.e. limit of $\mathcal{G}$-measurable functions is $\mathcal{G}$-measurable (after redefining on a null set), $Y$ is $\mathcal{G}$-measurable, hence $Y \in V$.

Now apply the $L^2$ projection theorem from Topic 27 §9 Proposition 3: there exists a unique element $\hat X \in V$ minimizing $\|X - \hat X\|_{L^2}$, characterized by the orthogonality condition

$\langle X - \hat X, Z \rangle_{L^2(P)} = \int (X - \hat X) Z \, dP = 0 \quad \text{for every } Z \in V.$

Specializing $Z = \mathbf{1}_A$ for $A \in \mathcal{G}$ (note $\mathbf{1}_A \in V$ since $A$ is $\mathcal{G}$-measurable), the orthogonality condition becomes

$\int_A (X - \hat X) \, dP = 0, \quad \text{i.e.,} \quad \int_A \hat X \, dP = \int_A X \, dP \quad \text{for every } A \in \mathcal{G}.$

This is exactly the defining property of conditional expectation. By the uniqueness in Definition 7, $\hat X = E[X \mid \mathcal{G}]$ a.e. So $E[X \mid \mathcal{G}]$ is the $L^2$ projection of $X$ onto $V$, which is the best predictor of $X$ given $\mathcal{G}$ in the mean-squared sense.

Let $(X, Y)$ be jointly normal with $\mu = (\mu_X, \mu_Y)$ and covariance matrix