Mean Value Theorem & Taylor Expansion

Let $f$ be continuous on $[a, b]$ and differentiable on $(a, b)$, with $f(a) = f(b)$. Then there exists $c \in (a, b)$ such that $f'(c) = 0$.

By the Extreme Value Theorem — which we proved in Completeness & Compactness using the Heine-Borel theorem — the continuous function $f$ on the compact set $[a, b]$ achieves its maximum value $M$ and its minimum value $m$.

Case 1: If $M = m$, then $f$ is constant on $[a, b]$, so $f'(c) = 0$ for every $c \in (a, b)$.

Case 2: If $M \neq m$, then since $f(a) = f(b)$, at least one of $M$ or $m$ is achieved at an interior point $c \in (a, b)$ (both endpoints have the same value, so if the max and the min were both at the endpoints, we’d have $M = f(a) = f(b) = m$, contradicting $M \neq m$).

Without loss of generality, suppose $f(c) = M$ with $c \in (a, b)$. Since $c$ is a local maximum and $f$ is differentiable at $c$, we have:

• For $h > 0$ sufficiently small: $\frac{f(c + h) - f(c)}{h} \leq 0$ (since $f(c + h) \leq f(c) = M$), so $f'(c) \leq 0$.
• For $h < 0$ sufficiently small: $\frac{f(c + h) - f(c)}{h} \geq 0$ (since $f(c + h) \leq f(c)$ and $h < 0$), so $f'(c) \geq 0$.

Together: $f'(c) = 0$.

We verify: $f(1) = 1 - 4 + 3 = 0$ and $f(3) = 9 - 12 + 3 = 0$, so $f(a) = f(b)$. The derivative is $f'(x) = 2x - 4$, which vanishes at $c = 2 \in (1, 3)$. Geometrically, the parabola dips below the $x$-axis and turns around at its vertex $x = 2$.

Each hypothesis in Rolle’s theorem is essential — removing any one produces a counterexample:

(i) Not continuous: Define $f$ on $[0, 1]$ by $f(x) = x$ for $0 \leq x < 1$ and $f(1) = 0$. Then $f(0) = f(1) = 0$, and $f$ is differentiable on $(0, 1)$ with $f'(x) = 1$ for all $x \in (0, 1)$ — so there is no $c$ with $f'(c) = 0$. The discontinuity at $x = 1$ breaks Rolle’s conclusion.

(ii) Not differentiable: $f(x) = |x|$ on $[-1, 1]$ has $f(-1) = f(1) = 1$, but $f'(0)$ does not exist (the left and right derivatives disagree — we saw this in The Derivative & Chain Rule, Example 3). The minimum of $|x|$ is at $x = 0$, exactly where differentiability fails.

(iii) Not $f(a) = f(b)$: The function $f(x) = x$ on $[0, 1]$ has $f'(x) = 1 \neq 0$ everywhere. Without the equal-endpoint condition, there’s no reason to expect a horizontal tangent.

Try toggling “Break hypotheses” in the explorer below to see each failure mode.

Let $f$ be continuous on $[a, b]$ and differentiable on $(a, b)$. Then there exists $c \in (a, b)$ such that

$f'(c) = \frac{f(b) - f(a)}{b - a}.$

In words: the instantaneous rate of change at some interior point $c$ equals the average rate of change over $[a, b]$.

Define the auxiliary function

$g(x) = f(x) - \frac{f(b) - f(a)}{b - a}(x - a).$

This is $f$ minus the secant line through $(a, f(a))$ and $(b, f(b))$.

Check the hypotheses of Rolle’s theorem:

• $g$ is continuous on $[a, b]$ (difference of continuous functions).
• $g$ is differentiable on $(a, b)$ (difference of differentiable functions).
• $g(a) = f(a) - 0 = f(a)$ and $g(b) = f(b) - \frac{f(b) - f(a)}{b - a}(b - a) = f(b) - (f(b) - f(a)) = f(a)$, so $g(a) = g(b)$.

By Rolle’s theorem, there exists $c \in (a, b)$ with $g'(c) = 0$. Since

$g'(x) = f'(x) - \frac{f(b) - f(a)}{b - a},$

we get $f'(c) = \frac{f(b) - f(a)}{b - a}$.

The secant slope is $\frac{f(2) - f(0)}{2 - 0} = \frac{8}{2} = 4$. We need $f'(c) = 3c^2 = 4$, so $c = \frac{2}{\sqrt{3}} \approx 1.155 \in (0, 2)$.

Drag the $a$ and $b$ sliders in the explorer below to see how the secant line and the parallel tangent change for different intervals.

If $f$ is differentiable on $(a, b)$ and $f'(x) = 0$ for all $x \in (a, b)$, then $f$ is constant on $(a, b)$.

For any $x_1, x_2 \in (a, b)$ with $x_1 < x_2$, the MVT gives

$f(x_2) - f(x_1) = f'(c)(x_2 - x_1) = 0 \cdot (x_2 - x_1) = 0$

for some $c \in (x_1, x_2)$. Hence $f(x_1) = f(x_2)$ for all $x_1, x_2 \in (a, b)$.

Let $f$ be differentiable on $(a, b)$.

• If $f'(x) > 0$ for all $x \in (a, b)$, then $f$ is strictly increasing on $(a, b)$.
• If $f'(x) < 0$ for all $x \in (a, b)$, then $f$ is strictly decreasing on $(a, b)$.

Suppose $f'(x) > 0$ on $(a, b)$ and take $x_1 < x_2$ in $(a, b)$. By the MVT, there exists $c \in (x_1, x_2)$ with

$f(x_2) - f(x_1) = f'(c)(x_2 - x_1).$

Since $f'(c) > 0$ and $x_2 - x_1 > 0$, we get $f(x_2) - f(x_1) > 0$, i.e., $f(x_1) < f(x_2)$. The decreasing case is symmetric.

If $f$ is differentiable on $(a, b)$ and $|f'(x)| \leq M$ for all $x \in (a, b)$, then

$|f(x) - f(y)| \leq M|x - y| \quad \text{for all } x, y \in (a, b).$

By the MVT, $|f(x) - f(y)| = |f'(c)| \cdot |x - y| \leq M|x - y|$.

The Lipschitz constant $M$ bounds how fast $f$ can change. In machine learning, the Lipschitz gradient condition $\|\nabla f(x) - \nabla f(y)\| \leq L\|x - y\|$ is the single most important regularity assumption in convergence proofs for gradient descent. It comes directly from applying the MVT (or its multivariable analog) to $\nabla f$: if $\nabla^2 f$ exists and $\|\nabla^2 f\| \leq L$, then the gradient is $L$-Lipschitz. This $L$ determines the maximum safe learning rate $\eta \leq 1/L$ and the convergence rate $O(1/k)$ for gradient descent on convex functions. (→ formalML: Gradient Descent)

Let $f$ and $g$ be continuous on $[a, b]$ and differentiable on $(a, b)$, with $g'(x) \neq 0$ for all $x \in (a, b)$. Then there exists $c \in (a, b)$ such that

$\frac{f'(c)}{g'(c)} = \frac{f(b) - f(a)}{g(b) - g(a)}.$

First note that $g(b) \neq g(a)$ (otherwise, Rolle’s theorem applied to $g$ would give $g'(c) = 0$ for some $c$, contradicting our hypothesis). Define

$h(x) = f(x)(g(b) - g(a)) - g(x)(f(b) - f(a)).$

Then $h$ is continuous on $[a, b]$, differentiable on $(a, b)$, and $h(a) = f(a)g(b) - g(a)f(b) = h(b)$. By Rolle’s theorem, there exists $c \in (a, b)$ with $h'(c) = 0$:

$f'(c)(g(b) - g(a)) = g'(c)(f(b) - f(a)).$

Since $g'(c) \neq 0$ and $g(b) - g(a) \neq 0$, dividing gives the result.

Suppose $f$ and $g$ are differentiable near $a$ (except possibly at $a$), with $g'(x) \neq 0$ near $a$. If

$\lim_{x \to a} f(x) = 0 \quad \text{and} \quad \lim_{x \to a} g(x) = 0,$

and the limit $\lim_{x \to a} \frac{f'(x)}{g'(x)} = L$ exists (finite or $\pm\infty$), then

$\lim_{x \to a} \frac{f(x)}{g(x)} = L.$

Define $f(a) = g(a) = 0$ so that $f$ and $g$ are continuous at $a$. For $x$ near $a$ with $x \neq a$, Cauchy’s MVT on $[a, x]$ (or $[x, a]$) gives a point $c_x$ between $a$ and $x$ with

$\frac{f(x)}{g(x)} = \frac{f(x) - f(a)}{g(x) - g(a)} = \frac{f'(c_x)}{g'(c_x)}.$

As $x \to a$, $c_x \to a$ (squeezed between $x$ and $a$), so $\frac{f'(c_x)}{g'(c_x)} \to L$.

$\lim_{x \to 0} \frac{\sin x}{x}$ is the $0/0$ form. Applying L’Hôpital’s:

$\lim_{x \to 0} \frac{\sin x}{x} = \lim_{x \to 0} \frac{\cos x}{1} = 1.$

We previously established this limit geometrically (a squeeze argument involving areas). Now we have a second, purely analytical proof via Cauchy’s MVT. Both proofs are valid; the geometric argument is more elementary, while L’Hôpital’s is more general.

L’Hôpital’s Rule is a sufficient condition, not a necessary one. The limit $\lim_{x \to 0} \frac{x^2 \sin(1/x)}{x} = \lim_{x \to 0} x \sin(1/x) = 0$ exists, but $\lim_{x \to 0} \frac{2x \sin(1/x) - \cos(1/x)}{1}$ does not exist (the $\cos(1/x)$ term oscillates). So L’Hôpital’s fails to apply, yet the original limit exists.

The rule also applies to the $\infty/\infty$ form, to limits as $x \to \infty$, and (via algebraic rearrangement) to $0 \cdot \infty$ and $\infty - \infty$ indeterminate forms.

Let $f$ be $n$-times differentiable at $a$. The degree-$n$ Taylor polynomial of $f$ centered at $a$ is

$T_n(x) = \sum_{k=0}^{n} \frac{f^{(k)}(a)}{k!}(x - a)^k = f(a) + f'(a)(x - a) + \frac{f''(a)}{2!}(x - a)^2 + \cdots + \frac{f^{(n)}(a)}{n!}(x - a)^n.$

The special case $a = 0$ gives the Maclaurin polynomial.

Among all polynomials $p$ of degree $\leq n$, the Taylor polynomial $T_n$ is the unique polynomial satisfying $p^{(k)}(a) = f^{(k)}(a)$ for $k = 0, 1, \ldots, n$. Equivalently, the error $f(x) - T_n(x)$ vanishes to order $n$ at $a$:

$\lim_{x \to a} \frac{f(x) - T_n(x)}{(x - a)^n} = 0.$

Since all derivatives of $e^x$ equal $e^x$, we have $f^{(k)}(0) = 1$ for all $k$:

$T_n(x) = 1 + x + \frac{x^2}{2!} + \frac{x^3}{3!} + \cdots + \frac{x^n}{n!}.$

As $n$ increases, $T_n$ approximates $e^x$ over a wider and wider interval. Try increasing the degree in the explorer below to watch the polynomial “hug” the exponential.

The derivatives of $\sin x$ cycle: $\sin, \cos, -\sin, -\cos, \sin, \ldots$ At $x = 0$: $f(0) = 0$, $f'(0) = 1$, $f''(0) = 0$, $f'''(0) = -1$, $f^{(4)}(0) = 0$, $f^{(5)}(0) = 1, \ldots$

Only odd terms survive:

$T_{2n+1}(x) = x - \frac{x^3}{3!} + \frac{x^5}{5!} - \frac{x^7}{7!} + \cdots + \frac{(-1)^n x^{2n+1}}{(2n+1)!}.$

The pattern reflects the odd symmetry of $\sin$ — an odd function’s Taylor series has only odd powers.

Let $f$ be $(n+1)$-times differentiable on an open interval containing $a$ and $x$. Then

$f(x) = T_n(x) + R_n(x), \quad \text{where} \quad R_n(x) = \frac{f^{(n+1)}(c)}{(n+1)!}(x - a)^{n+1}$

for some $c$ between $a$ and $x$.

We prove this by applying Rolle’s theorem to a carefully constructed auxiliary function. Fix $x \neq a$ and define

$\phi(t) = f(t) - T_n(t) - K(t - a)^{n+1},$

where $T_n(t) = \sum_{k=0}^{n} \frac{f^{(k)}(a)}{k!}(t - a)^k$ is the degree-$n$ Taylor polynomial of $f$ centered at $a$ (evaluated at the variable $t$), and $K$ is a constant chosen so that $\phi(x) = 0$.

First note that $T_n(a) = f(a)$, so $\phi(a) = f(a) - f(a) - 0 = 0$.

Next, the condition $\phi(x) = 0$ gives

$0 = f(x) - T_n(x) - K(x - a)^{n+1}, \quad \text{so} \quad K = \frac{f(x) - T_n(x)}{(x - a)^{n+1}}.$

Our goal is to show that $K = \frac{f^{(n+1)}(c)}{(n+1)!}$ for some $c$ between $a$ and $x$.

Since $\phi(a) = \phi(x) = 0$, Rolle’s theorem gives some $c_1$ between $a$ and $x$ with $\phi'(c_1) = 0$. We differentiate $\phi$ a total of $n+1$ times. At each stage, the polynomial terms from $T_n(t)$ lose one degree, and after $n+1$ differentiations the Taylor polynomial terms vanish entirely:

$\phi^{(n+1)}(t) = f^{(n+1)}(t) - K \cdot (n+1)!$

(since $(t - a)^{n+1}$ differentiates $n+1$ times to $(n+1)!$, and $T_n(t)$ is a degree-$n$ polynomial whose $(n+1)$-th derivative is $0$).

By applying Rolle’s theorem successively — $\phi(a) = \phi(x) = 0$ gives a zero of $\phi'$, which combined with $\phi'(a) = 0$ gives a zero of $\phi''$, and so on — we obtain a point $c$ between $a$ and $x$ with $\phi^{(n+1)}(c) = 0$. Setting this to zero:

$0 = f^{(n+1)}(c) - K(n+1)! \quad \Rightarrow \quad K = \frac{f^{(n+1)}(c)}{(n+1)!}.$

Combining with the expression $K = \frac{f(x) - T_n(x)}{(x - a)^{n+1}}$ gives the Lagrange remainder:

$f(x) - T_n(x) = \frac{f^{(n+1)}(c)}{(n+1)!}(x - a)^{n+1}.$

Note: The base case $n = 0$ gives $f(x) = f(a) + f'(c)(x - a)$, which is exactly the Mean Value Theorem. Taylor’s theorem is the higher-order generalization of the MVT.

An alternative expression for the remainder is

$R_n(x) = \int_a^x \frac{f^{(n+1)}(t)}{n!}(x - t)^n \, dt.$

This avoids the unknown point $c$ and is sometimes more useful for estimation. The integral form is proved using integration by parts in The Riemann Integral & FTC, §9.

For $f(x) = e^x$ with $a = 0$: all derivatives are $e^x$, so $f^{(n+1)}(c) = e^c$ for some $c$ between $0$ and $x$. The Lagrange remainder is

$|R_n(x)| = \frac{e^c}{(n+1)!}|x|^{n+1} \leq \frac{e^{|x|}}{(n+1)!}|x|^{n+1}.$

For any fixed $x$, $R_n(x) \to 0$ as $n \to \infty$ because the factorial $(n+1)!$ grows faster than any power $|x|^{n+1}$. This proves that $e^x = \sum_{k=0}^{\infty} \frac{x^k}{k!}$ converges for all $x \in \mathbb{R}$.

For $f(x) = \sin x$ with $a = 0$: all derivatives of $\sin$ are bounded by $1$ in absolute value, so

$|R_n(x)| \leq \frac{1}{(n+1)!}|x|^{n+1} \to 0 \quad \text{as } n \to \infty$

for any fixed $x$. The Maclaurin series of $\sin x$ converges to $\sin x$ for all $x$.

Define

$f(x) = \begin{cases} e^{-1/x^2} & \text{if } x \neq 0, \\ 0 & \text{if } x = 0. \end{cases}$

This function is $C^\infty$ (infinitely differentiable) everywhere. At $x = 0$, one can show by induction (using L’Hôpital’s Rule from §5 at each step) that $f^{(k)}(0) = 0$ for all $k \geq 0$. So the Taylor series at $0$ is

$T(x) = 0 + 0 + 0 + \cdots = 0,$

but $f(x) > 0$ for all $x \neq 0$. The Taylor series converges (to $0$), but it does not converge to $f(x)$. The function “flattens” against zero so aggressively at $x = 0$ that no polynomial can detect the non-zero values away from the origin.

A function $f$ is analytic at $a$ if its Taylor series at $a$ converges to $f(x)$ in some neighborhood of $a$: there exists $r > 0$ such that $f(x) = \sum_{k=0}^{\infty} \frac{f^{(k)}(a)}{k!}(x - a)^k$ for all $|x - a| < r$.

In practice, loss functions in ML are typically compositions of analytic functions (exponentials, logarithms, polynomials), so the Taylor expansion is a reliable local model. ReLU is piecewise linear (hence piecewise analytic). The distinction between smooth and analytic matters more in theory — the existence of smooth bump functions (which are $C^\infty$ with compact support and are not analytic) is essential for partition-of-unity arguments in differential geometry. (→ formalML: Smooth Manifolds)