Uniform Convergence

Overview & Motivation

You’re training a neural network with increasing width: $f_1, f_2, f_4, f_8, \ldots$ neurons per hidden layer. Each $f_n$ is a continuous function — compositions of affine maps and continuous activations like ReLU or sigmoid. As $n \to \infty$, does $f_n$ converge to some function $f$? And if so, is $f$ still continuous? Can you integrate $f$ by integrating the $f_n$? Can you differentiate $f$ by differentiating the $f_n$?

The answers depend on how $f_n$ converges — and the wrong notion of convergence gives the wrong answers to all three questions.

In Sequences, Limits & Convergence, we studied convergence of sequences of numbers: given a sequence $a_1, a_2, a_3, \ldots$ in $\mathbb{R}$, does $a_n \to L$? In Epsilon-Delta & Continuity, we studied continuity of individual functions: is $f$ continuous at $a$? In Completeness & Compactness, we studied structural properties that guarantee limits exist and optima are attained. Now we combine all three: when does a sequence of functions converge, and when does the limit inherit the properties of the functions themselves?

The answer will require a new notion of convergence — uniform convergence — that is strictly stronger than the naive “pointwise” approach. The distinction between these two notions is not a technicality. It is the mathematical backbone of generalization theory in machine learning.

Pointwise Convergence of Function Sequences

The simplest way to define convergence of a function sequence is to check convergence at each point separately.

The crucial phrase is “which may depend on both $x$ and $\varepsilon$.” We write $N = N(x, \varepsilon)$ to emphasize this dependence. At each fixed $x$, we’re just asking for ordinary sequence convergence — the $\varepsilon$-$N$ definition from Sequences, Limits & Convergence. But the $N$ required might be very different at different points $x$.

The canonical example makes this concrete:

Notice what happened: each $f_n(x) = x^n$ is continuous on $[0, 1]$, but the pointwise limit $f$ has a jump discontinuity at $x = 1$. The limit of continuous functions is not continuous.

A second example where things go right: $g_n(x) = x^n / n$ on $[0, 1]$ also converges pointwise to $f \equiv 0$ (the same limit for $x \in [0,1)$; at $x = 1$, $g_n(1) = 1/n \to 0$). And this time the limit is continuous. What’s different?

The difference is the convergence rate. For $f_n(x) = x^n$, points near $x = 1$ require enormous $N$ to get close to the limit — $N(x, \varepsilon) = \lceil \log \varepsilon / \log x \rceil \to \infty$ as $x \to 1^-$. For $g_n(x) = x^n/n$, the sup-norm $\|g_n\|_\infty = 1/n$ is independent of $x$, so a single $N$ works everywhere.

The Three Pathologies of Pointwise Convergence

Pointwise convergence fails to preserve the three fundamental properties of functions:

1. Continuity failure. We’ve just seen this: $f_n(x) = x^n$ is continuous for every $n$, but $f_n \to f$ pointwise where $f$ is discontinuous. The limit of continuous functions need not be continuous under pointwise convergence.

2. Integration failure. Consider the “moving bump” sequence $f_n(x) = n^2 x (1 - x)^n$ on $[0, 1]$. Each $f_n$ has a bump near $x = 0$ that grows taller and narrower. One can verify that $\int_0^1 f_n(x)\,dx = \frac{n^2}{(n+1)(n+2)} \to 1$ as $n \to \infty$. But $f_n(x) \to 0$ for each fixed $x > 0$ (and $f_n(0) = 0$), so the pointwise limit is $f \equiv 0$, and $\int_0^1 f(x)\,dx = 0$. We get

$\lim_{n \to \infty} \int_0^1 f_n(x)\,dx = 1 \neq 0 = \int_0^1 \left(\lim_{n \to \infty} f_n(x)\right) dx.$

The limit and integral do not commute.

3. Differentiation failure. Let $f_n(x) = \frac{\sin(nx)}{\sqrt{n}}$ on $\mathbb{R}$. Then $|f_n(x)| \leq 1/\sqrt{n} \to 0$ uniformly, so $f_n \to 0$ everywhere. But $f_n'(x) = \sqrt{n}\cos(nx)$, and $|f_n'(0)| = \sqrt{n} \to \infty$. The derivatives of the $f_n$ don’t converge at all, even though the $f_n$ themselves converge nicely.

Uniform Convergence — The Fix

If we require the convergence rate to be the same at every $x$ — that is, the $N$ depends only on $\varepsilon$, not on $x$ — then all three pathologies disappear.

The geometric picture is the ε-band characterization: $f_n \to f$ uniformly if and only if for every $\varepsilon > 0$, eventually the entire graph of $f_n$ lies within an $\varepsilon$-band around the graph of $f$. Not just at each point separately, but everywhere at once.

The comparison is now sharp:

• Pointwise: “for each $x$, the vertical distance $|f_n(x) - f(x)| \to 0$.”
• Uniform: “the maximum vertical distance over all $x$ simultaneously $\to 0$.”

Use the explorer below to feel the difference. Drag the $n$ slider and watch whether the function’s graph stays inside the ε-band. The moment when $x^n$ escapes the band near $x = 1$ — but $x^n/n$ stays inside — is the difference between pointwise and uniform convergence.

Sequence:f_n(x) = x^n on [0, 1]g_n(x) = x^n / n on [0, 1]h_n(x) = sin(nx)/(n+1) on [0, π]p_n(x) = nx(1-x)^n on [0, 1]q_n(x) = x/(1+nx) on [0, 1]Show ε-band

n = 5ε = 0.100

‖f_n − f‖_∞ = 0.9900✗ Escapes ε-band (not uniform at this n)Pointwise only — NOT uniform

||f_n - f||_∞ → 1 (never → 0) · Analogous to an overfit model converging fast in-distribution but failing at the boundary.

The Uniform Limit Theorem

The most important theorem in this topic: uniform convergence preserves continuity.

The key insight: uniform convergence provides a single $N$ that simultaneously controls the first and third terms for all $x$. Pointwise convergence would give different $N$ for different $x$, and we couldn’t fix $N$ before choosing $\delta$.

Mode:Uniform convergencePointwise only

n = 10Click on the graph to probe continuity at a point

Uniform convergence (x^n/n → 0): the limit is continuous. The ε/3 argument works because one N controls all x.

Interchange Theorems — Integration and Differentiation

Uniform convergence enables two crucial interchange theorems, but with different hypotheses.

The proof is clean and direct — the sup-norm bound converts a pointwise inequality into a uniform one, and the integral of a uniform bound is just the bound times the interval length.

The differentiation interchange is harder and requires a stronger hypothesis:

The Cauchy Criterion for Uniform Convergence

Just as numerical sequences have a Cauchy criterion that certifies convergence without knowing the limit (Sequences, Limits & Convergence, Definition 6), function sequences have an analogous criterion.

Series of Functions and the Weierstrass M-Test

A series of functions $\sum_{k=1}^\infty g_k(x)$ is just a sequence of partial sums. The Weierstrass M-test provides a practical, checkable criterion for uniform convergence.

Series:∑ sin(kx)/k²∑ cos(kx)/k!∑ sin(kx)/kShow M_k bounds

n = 5 terms

Actual ‖S − S_n‖_∞ ≈ 0.10488M-test bound ≈ 0.17634✓ M-test passes

M_k = 1/k² (∑ = π²/6) · Left: partial sums S_n(x) vs. full sum. Right: M_k bound bars (blue = included, gray = tail).

Equicontinuity and the Arzelà-Ascoli Theorem

The Heine-Borel theorem from Completeness & Compactness says that a subset of $\mathbb{R}^n$ is compact if and only if it is closed and bounded. What is the analog for subsets of the function space $C([a,b])$? Boundedness alone is not enough — we need a condition that controls how much the functions in the family can oscillate. That condition is equicontinuity.

Compare this with uniform continuity from Epsilon-Delta & Continuity (Definition 9): uniform continuity says a single function has one $\delta$ that works at every point; equicontinuity says a family of functions has one $\delta$ that works for every function and every point simultaneously.

Family:Bounded Lipschitz (equicontinuous)Increasing oscillation (NOT equicontinuous)

Generate FamilyExtract Subsequence

Equicontinuous: ✓Pointwise bounded: ✓Arzelà-Ascoli applies: ✓Lipschitz bound: 3

Click on the graph to probe equicontinuity at a point. Blue curves = extracted subsequence. Dashed red = limit approximation.

Connections to ML

Uniform convergence is not merely a theoretical curiosity that happens to appear in machine learning. It is the mathematical foundation of generalization theory — the question of why models that perform well on training data also perform well on unseen data.

Uniform convergence of empirical processes

Given a hypothesis class $\mathcal{H}$ (e.g., the set of all neural networks with a fixed architecture), the empirical risk of a hypothesis $h \in \mathcal{H}$ on $n$ training examples is

$\hat{R}_n(h) = \frac{1}{n}\sum_{i=1}^n \ell(h(x_i), y_i),$

while the true risk is $R(h) = \mathbb{E}[\ell(h(X), Y)]$. The law of large numbers guarantees that for any fixed $h$, $\hat{R}_n(h) \to R(h)$ as $n \to \infty$. This is pointwise convergence — one hypothesis at a time.

But learning requires more: we need $\hat{R}_n(h)$ to be close to $R(h)$ for all $h \in \mathcal{H}$ simultaneously, so that the hypothesis $\hat{h}$ that minimizes $\hat{R}_n$ also approximately minimizes $R$. This is uniform convergence:

$\sup_{h \in \mathcal{H}} \left|\hat{R}_n(h) - R(h)\right| \to 0.$

This is exactly the sup-norm $\|\hat{R}_n - R\|_\infty \to 0$ over the function class $\mathcal{H}$. → formalML PAC Learning

The Glivenko-Cantelli theorem

The oldest and most fundamental uniform convergence result in statistics: the empirical CDF $\hat{F}_n(x) = \frac{1}{n}\sum_{i=1}^n \mathbf{1}_{X_i \leq x}$ converges uniformly to the true CDF $F(x)$:

$\|\hat{F}_n - F\|_\infty = \sup_{x \in \mathbb{R}} |\hat{F}_n(x) - F(x)| \to 0 \quad \text{almost surely.}$

The Dvoretzky-Kiefer-Wolfowitz (DKW) inequality gives the rate: $P\left(\|\hat{F}_n - F\|_\infty > t\right) \leq 2e^{-2nt^2}$, yielding $\|\hat{F}_n - F\|_\infty = O(1/\sqrt{n})$. The Kolmogorov-Smirnov test is built on this sup-norm. → formalML Concentration Inequalities

VC dimension and complexity control

The VC dimension $d$ of a hypothesis class $\mathcal{H}$ quantifies how “complex” $\mathcal{H}$ is — it is the largest number of points that $\mathcal{H}$ can shatter (classify in all $2^d$ possible ways). The fundamental theorem of statistical learning: $\mathcal{H}$ is PAC-learnable if and only if $\mathcal{H}$ has finite VC dimension. The generalization bound

$\sup_{h \in \mathcal{H}} |\hat{R}_n(h) - R(h)| \leq O\left(\sqrt{\frac{d \log n}{n}}\right)$

is a rate of uniform convergence — it tells you how fast the sup-norm between empirical and true risk shrinks with sample size. → formalML PAC Learning

Arzelà-Ascoli and neural network approximation

The universal approximation theorem says that neural networks are dense in $C([a,b])$ under the sup-norm: for any continuous $f$ and any $\varepsilon > 0$, there exists a neural network $g$ with $\|f - g\|_\infty < \varepsilon$. Arzelà-Ascoli constrains which families of networks are compact (and hence tractable): a family with bounded Lipschitz constants is equicontinuous, and equicontinuity + boundedness gives compactness. This is why Lipschitz constraints (spectral normalization, gradient penalty) appear in generative models — they enforce the compactness that makes optimization well-posed.

Computational Notes

Uniform convergence can be verified computationally by evaluating the sup-norm on a dense grid.

Computing the sup-norm. Given a function sequence $f_n$ and its limit $f$, we approximate $\|f_n - f\|_\infty$ by sampling:

import numpy as np

def sup_norm(fn, f, domain, grid_size=1000):
    """Approximate ||fn - f||_∞ on domain."""
    x = np.linspace(domain[0], domain[1], grid_size)
    return np.max(np.abs(fn(x) - f(x)))

# x^n on [0, 1]: not uniform (sup norm → 1)
for n in [10, 50, 100, 500]:
    fn = lambda x, n=n: x**n
    f = lambda x: np.where(x < 1, 0.0, 1.0)
    print(f"n={n:3d}: ||f_n - f||_∞ = {sup_norm(fn, f, (0, 1)):.6f}")

# x^n/n on [0, 1]: uniform (sup norm = 1/n → 0)
for n in [10, 50, 100, 500]:
    gn = lambda x, n=n: x**n / n
    g = lambda x: np.zeros_like(x)
    print(f"n={n:3d}: ||g_n - g||_∞ = {sup_norm(gn, g, (0, 1)):.6f}")

Verifying the integration interchange. Compare $\int f_n$ with $\int f$:

from scipy import integrate

def check_interchange(fn, f, domain, n_values):
    """Compare lim ∫f_n with ∫(lim f_n)."""
    a, b = domain
    integral_f = integrate.quad(f, a, b)[0]
    for n in n_values:
        integral_fn = integrate.quad(lambda x: fn(x, n), a, b)[0]
        print(f"n={n:3d}: ∫f_n = {integral_fn:.6f}, ∫f = {integral_f:.6f}, "
              f"gap = {abs(integral_fn - integral_f):.6f}")

# Uniform case: gap → 0
check_interchange(
    fn=lambda x, n: x**n / n,
    f=lambda x: 0.0,
    domain=(0, 1),
    n_values=[5, 20, 100]
)

Implementing the Weierstrass M-test. Check the bound and compare with the actual tail error:

def weierstrass_m_test(term, Mk, domain, n_terms, max_k=200, grid=1000):
    """Apply the M-test: compare actual tail error with M-bound."""
    x = np.linspace(domain[0], domain[1], grid)
    partial = sum(term(x, k) for k in range(1, n_terms + 1))
    full = sum(term(x, k) for k in range(1, max_k + 1))
    actual_error = np.max(np.abs(full - partial))
    m_bound = sum(Mk(k) for k in range(n_terms + 1, max_k + 1))
    print(f"n={n_terms}: actual ||S-S_n||_∞ = {actual_error:.6f}, "
          f"M-bound = {m_bound:.6f}")

# ∑ sin(kx)/k²
weierstrass_m_test(
    term=lambda x, k: np.sin(k * x) / k**2,
    Mk=lambda k: 1 / k**2,
    domain=(0, 2 * np.pi),
    n_terms=10
)

Computing the Kolmogorov-Smirnov statistic. The sup-norm between empirical and true CDF:

def kolmogorov_smirnov(samples, cdf):
    """Compute ||F_hat_n - F||_∞."""
    n = len(samples)
    sorted_samples = np.sort(samples)
    ecdf = np.arange(1, n + 1) / n
    sup_diff = np.max(np.abs(ecdf - cdf(sorted_samples)))
    return sup_diff

# Standard normal: Glivenko-Cantelli in action
from scipy.stats import norm
for n in [100, 1000, 10000]:
    samples = np.random.randn(n)
    ks = kolmogorov_smirnov(samples, norm.cdf)
    dkw_bound = np.sqrt(np.log(2 / 0.05) / (2 * n))
    print(f"n={n:5d}: KS = {ks:.4f}, DKW 95% bound = {dkw_bound:.4f}")

Connections & Further Reading

Within formalCalculus

• Sequences, Limits & Convergence — the ε-N convergence framework for numerical sequences. Pointwise convergence of $f_n(x)$ at each fixed $x$ is literally a collection of numerical sequence convergence problems. The Cauchy criterion for uniform convergence (Proposition 2) is the function-space analog of the Cauchy sequence criterion. The Bolzano-Weierstrass theorem is the engine of the diagonal argument in the Arzelà-Ascoli proof.
• Epsilon-Delta & Continuity — the ε-δ continuity framework used in the Uniform Limit Theorem proof (the ε/3 argument). Uniform continuity from Topic 2 is the single-function version of the equicontinuity concept. The Lipschitz hierarchy reappears: a family with bounded Lipschitz constants is equicontinuous.
• Completeness & Compactness — the Heine-Borel theorem is the finite-dimensional analog of Arzelà-Ascoli. Equicontinuity + pointwise boundedness plays the role that closed + bounded plays in Heine-Borel. Completeness of $\mathbb{R}$ underlies the Cauchy criterion proof.
• Series Convergence & Tests — the Weierstrass M-test applied to specific series; absolute vs. conditional convergence for function series.
• Power Series & Taylor Series — power series converge uniformly on compact subsets of the radius of convergence; term-by-term differentiation and integration are direct applications of the interchange theorems.
• Metric Spaces & Topology — $C([a,b])$ with the sup-norm is proved complete, and the Arzelà–Ascoli theorem is proved in full via the diagonal subsequence argument, closing both promises from this topic.
• Normed & Banach Spaces — $C([a,b])$ is a Banach space (complete normed vector space); completeness is the Cauchy criterion for uniform convergence.
• Approximation Theory & Convergence Rates — Stone-Weierstrass theorem (polynomials are dense in $C([a,b])$ under the sup-norm); rates of uniform approximation.

Forward to formalML

• formalML PAC Learning — uniform convergence of empirical risk to true risk; VC dimension controls the rate.
• formalML Concentration Inequalities — Glivenko-Cantelli, DKW inequality, uniform bounds over function classes.
• formalML Measure-Theoretic Probability — modes of convergence in probability extend the pointwise/uniform distinction to random functions.