The Derivative & Chain Rule

Let $f$ be defined on an open interval containing $a$. The derivative of $f$ at $a$ is

$f'(a) = \lim_{h \to 0} \frac{f(a+h) - f(a)}{h},$

provided this limit exists. Equivalently, using the ε-δ framework from Epsilon-Delta & Continuity: for every $\varepsilon > 0$, there exists $\delta > 0$ such that

$0 < |h| < \delta \implies \left|\frac{f(a+h) - f(a)}{h} - f'(a)\right| < \varepsilon.$

When this limit exists, we say $f$ is differentiable at $a$. The function $f': x \mapsto f'(x)$, defined at every point where the limit exists, is the derivative function of $f$.

The left derivative and right derivative of $f$ at $a$ are

$f'_-(a) = \lim_{h \to 0^-} \frac{f(a+h) - f(a)}{h}, \qquad f'_+(a) = \lim_{h \to 0^+} \frac{f(a+h) - f(a)}{h}.$

The derivative $f'(a)$ exists if and only if both one-sided derivatives exist and $f'_-(a) = f'_+(a)$.

We compute $f'(3)$ directly from the definition:

$f'(3) = \lim_{h \to 0} \frac{(3+h)^2 - 9}{h} = \lim_{h \to 0} \frac{9 + 6h + h^2 - 9}{h} = \lim_{h \to 0} \frac{6h + h^2}{h} = \lim_{h \to 0} (6 + h) = 6.$

Every step is algebra followed by a limit. The cancellation of $h$ in the numerator and denominator is the key move — it removes the $\frac{0}{0}$ indeterminate form and leaves a polynomial in $h$ that we can evaluate at $h = 0$.

For $f(x) = \sqrt{x}$ at $a > 0$:

$f'(a) = \lim_{h \to 0} \frac{\sqrt{a+h} - \sqrt{a}}{h}.$

The direct approach gives $\frac{0}{0}$, so we rationalize — multiply numerator and denominator by the conjugate $\sqrt{a+h} + \sqrt{a}$:

$\frac{\sqrt{a+h} - \sqrt{a}}{h} \cdot \frac{\sqrt{a+h} + \sqrt{a}}{\sqrt{a+h} + \sqrt{a}} = \frac{(a+h) - a}{h(\sqrt{a+h} + \sqrt{a})} = \frac{1}{\sqrt{a+h} + \sqrt{a}}.$

As $h \to 0$, this converges to $\frac{1}{2\sqrt{a}}$. So $f'(a) = \frac{1}{2\sqrt{a}}$, which is defined for $a > 0$ — the domain of the derivative is smaller than the domain of $f$ (which includes $a = 0$). At $a = 0$, the difference quotient $\frac{\sqrt{h}}{h} = \frac{1}{\sqrt{h}} \to \infty$, so $f$ is not differentiable there (vertical tangent).

$f$ is differentiable at $a$ with derivative $f'(a)$ if and only if there exists a number $L$ such that

$f(a+h) = f(a) + Lh + r(h),$

where $\lim_{h \to 0} \frac{r(h)}{h} = 0$. The number $L$ is unique and equals $f'(a)$. The remainder $r(h)$ is said to be “little-o of $h$”, written $r(h) = o(h)$.

In one dimension, a linear map $L: \mathbb{R} \to \mathbb{R}$ is multiplication by a scalar: $L(h) = ch$. The derivative $f'(a)$ is that scalar. In $\mathbb{R}^n$, the derivative at $a$ will be a matrix (the Jacobian $J_f(a)$), and “multiplication by a scalar” becomes “multiplication by a matrix.” The single-variable derivative is the $1 \times 1$ case of the Jacobian.

If $f$ is differentiable at $a$, then $f$ is continuous at $a$.

We need to show that $\lim_{h \to 0} f(a+h) = f(a)$, which is continuity at $a$ (Epsilon-Delta & Continuity, Definition 5). Write

$f(a+h) - f(a) = \frac{f(a+h) - f(a)}{h} \cdot h.$

As $h \to 0$, the first factor converges to $f'(a)$ (by differentiability) and the second factor converges to $0$. By the algebra of limits (Sequences, Limits & Convergence, Theorem 3), the product converges to $f'(a) \cdot 0 = 0$. So $\lim_{h \to 0} [f(a+h) - f(a)] = 0$, which means $\lim_{h \to 0} f(a+h) = f(a)$. ∎

Continuity does not imply differentiability. The function $f(x) = |x|$ is continuous everywhere — including at $x = 0$ — but is not differentiable at $0$. This is not a pathological edge case: every ReLU unit in a neural network has a corner at $x = 0$ with the same non-differentiability as $|x|$. In practice, ML frameworks define the “derivative” at the kink by convention (typically $0$ or $1$), which works because the set of inputs landing exactly on the kink has measure zero.

The left derivative at $0$:

$f'_-(0) = \lim_{h \to 0^-} \frac{|0+h| - |0|}{h} = \lim_{h \to 0^-} \frac{|h|}{h} = \lim_{h \to 0^-} \frac{-h}{h} = -1.$

The right derivative at $0$:

$f'_+(0) = \lim_{h \to 0^+} \frac{|h|}{h} = \lim_{h \to 0^+} \frac{h}{h} = 1.$

Since $f'_-(0) = -1 \neq 1 = f'_+(0)$, the derivative $f'(0)$ does not exist. Geometrically, the graph of $|x|$ has a corner at $0$ — there is no single tangent line, because the secant from the left settles on slope $-1$ while the secant from the right settles on slope $+1$.

If $f$ and $g$ are differentiable at $a$, then $f + g$ is differentiable at $a$ and

$(f + g)'(a) = f'(a) + g'(a).$

$\frac{(f+g)(a+h) - (f+g)(a)}{h} = \frac{f(a+h) - f(a)}{h} + \frac{g(a+h) - g(a)}{h}.$

Taking the limit as $h \to 0$ and applying the sum rule for limits (Sequences, Limits & Convergence, Theorem 3), we get $f'(a) + g'(a)$. ∎

If $f$ is differentiable at $a$ and $c \in \mathbb{R}$, then $(cf)'(a) = c \cdot f'(a)$.

If $f$ and $g$ are differentiable at $a$, then $fg$ is differentiable at $a$ and

$(fg)'(a) = f'(a)\,g(a) + f(a)\,g'(a).$

The key is the add-and-subtract trick. Write

$f(a+h)\,g(a+h) - f(a)\,g(a) = f(a+h)\,g(a+h) - f(a+h)\,g(a) + f(a+h)\,g(a) - f(a)\,g(a).$

Factor:

$= f(a+h)\bigl[g(a+h) - g(a)\bigr] + g(a)\bigl[f(a+h) - f(a)\bigr].$

Divide by $h$:

$\frac{(fg)(a+h) - (fg)(a)}{h} = f(a+h) \cdot \frac{g(a+h) - g(a)}{h} + g(a) \cdot \frac{f(a+h) - f(a)}{h}.$

As $h \to 0$: the first term converges to $f(a) \cdot g'(a)$ (using Theorem 1: $f$ is differentiable hence continuous, so $f(a+h) \to f(a)$), and the second term converges to $g(a) \cdot f'(a)$. By the algebra of limits, the sum converges to $f(a)\,g'(a) + g(a)\,f'(a)$. ∎

If $f$ and $g$ are differentiable at $a$ and $g(a) \neq 0$, then $f/g$ is differentiable at $a$ and

$\left(\frac{f}{g}\right)'(a) = \frac{f'(a)\,g(a) - f(a)\,g'(a)}{g(a)^2}.$

Write the difference quotient:

$\frac{\frac{f(a+h)}{g(a+h)} - \frac{f(a)}{g(a)}}{h} = \frac{f(a+h)\,g(a) - f(a)\,g(a+h)}{h \cdot g(a+h) \cdot g(a)}.$

Add and subtract $f(a)\,g(a)$ in the numerator:

$= \frac{[f(a+h) - f(a)]\,g(a) - f(a)\,[g(a+h) - g(a)]}{h \cdot g(a+h) \cdot g(a)}.$

Split the fraction and take the limit. The numerator converges to $f'(a)\,g(a) - f(a)\,g'(a)$. The denominator converges to $g(a)^2$ (since $g$ is continuous at $a$ by Theorem 1, so $g(a+h) \to g(a)$, and $g(a) \neq 0$ ensures the limit is well-defined). ∎

For $f(x) = x^n$ with $n$ a positive integer:

$\frac{(a+h)^n - a^n}{h} = \frac{1}{h}\sum_{k=0}^{n}\binom{n}{k}a^{n-k}h^k - \frac{a^n}{h} = \sum_{k=1}^{n}\binom{n}{k}a^{n-k}h^{k-1}.$

As $h \to 0$, every term with $k \geq 2$ vanishes (it contains $h$ to a positive power), leaving only the $k = 1$ term:

$f'(a) = \binom{n}{1}a^{n-1} = n\,a^{n-1}.$

This is the power rule: $\frac{d}{dx}x^n = nx^{n-1}$.

If $g$ is differentiable at $a$ and $f$ is differentiable at $g(a)$, then $f \circ g$ is differentiable at $a$ and

$(f \circ g)'(a) = f'(g(a)) \cdot g'(a).$

We use the linear approximation characterization (Proposition 1). Since $f$ is differentiable at $g(a)$, we can write

$f(g(a) + k) = f(g(a)) + f'(g(a)) \cdot k + \varphi(k) \cdot k,$

where $\varphi(k) \to 0$ as $k \to 0$ (and we define $\varphi(0) = 0$).

The subtle point: A naive proof would divide by $g(a+h) - g(a)$ and recombine, but this fails when $g(a+h) = g(a)$ for $h \neq 0$ (which can happen — for example, $g(x) = x^2\sin(1/x)$ with $g(0) = 0$). The linear approximation approach avoids division entirely.

Set $k = g(a+h) - g(a)$. Then

$f(g(a+h)) = f(g(a) + k) = f(g(a)) + f'(g(a)) \cdot k + \varphi(k) \cdot k.$

Subtract $f(g(a))$ and divide by $h$:

$\frac{f(g(a+h)) - f(g(a))}{h} = f'(g(a)) \cdot \frac{k}{h} + \varphi(k) \cdot \frac{k}{h} = \bigl[f'(g(a)) + \varphi(k)\bigr] \cdot \frac{g(a+h) - g(a)}{h}.$

As $h \to 0$: the factor $\frac{g(a+h) - g(a)}{h} \to g'(a)$ by differentiability of $g$, and $k = g(a+h) - g(a) \to 0$ (by continuity of $g$, which follows from differentiability via Theorem 1), so $\varphi(k) \to 0$. By the algebra of limits:

$(f \circ g)'(a) = \bigl[f'(g(a)) + 0\bigr] \cdot g'(a) = f'(g(a)) \cdot g'(a). \quad \text{∎}$

Let $g(x) = x^2$ and $f(u) = \sin(u)$, so $f(g(x)) = \sin(x^2)$.

• Inner derivative: $g'(x) = 2x$
• Outer derivative: $f'(u) = \cos(u)$, evaluated at $u = g(x) = x^2$: $f'(g(x)) = \cos(x^2)$
• Chain rule: $(f \circ g)'(x) = \cos(x^2) \cdot 2x$

At $x = a$: $\frac{d}{dx}\sin(x^2)\big|_{x=a} = 2a\cos(a^2)$.

This is a composition of three functions: $h_1(x) = x^2$, $h_2(u) = \sin(u)$, $h_3(v) = e^v$. The chain rule applies iteratively:

$\frac{d}{dx}e^{\sin(x^2)} = e^{\sin(x^2)} \cdot \cos(x^2) \cdot 2x.$

Three derivatives, multiplied in order from outermost to innermost. This is a three-layer “network” — the chain rule applies at each layer. In a neural network with $L$ layers, we would multiply $L$ local derivatives.

Each derivative $f'(a)$ is a linear map $\mathbb{R} \to \mathbb{R}$ (multiplication by $f'(a)$). The chain rule says: the derivative of a composition is the composition of the derivatives. In one dimension, composing two linear maps means multiplying two scalars: $f'(g(a)) \cdot g'(a)$.

In $\mathbb{R}^n$, “multiplication” becomes matrix multiplication:

$J_{f \circ g}(a) = J_f(g(a)) \cdot J_g(a),$

where $J_f$ and $J_g$ are Jacobian matrices. The chain rule is functorial — it respects the compositional structure of functions. This perspective is the starting point for differential geometry, where the derivative is the pushforward map between tangent spaces. (→ formalML: Smooth Manifolds)

The second derivative of $f$ at $a$ is $f''(a) = (f')'(a)$, provided $f'$ is differentiable at $a$. Inductively, the $n$-th derivative is $f^{(n)}(a) = (f^{(n-1)})'(a)$.

Notation: $f''$ or $\frac{d^2f}{dx^2}$ for the second derivative; $f^{(n)}$ or $\frac{d^nf}{dx^n}$ for the $n$-th derivative.

The sign of $f''(a)$ determines the concavity of $f$ at $a$:

• $f''(a) > 0$: $f$ is concave up at $a$ — the graph curves upward, and the tangent line lies below the graph. This is the 1D condition for a local minimum.
• $f''(a) < 0$: $f$ is concave down at $a$ — the graph curves downward, and the tangent line lies above the graph. This is the 1D condition for a local maximum.

In higher dimensions, the second derivative becomes the Hessian matrix $H_f(a)$, and “concave up” becomes “positive definite” — the multi-dimensional criterion for a local minimum. This is the foundation of Newton’s method and second-order optimization. (The Hessian & Second-Order Analysis (coming soon))

ReLU$(x) = \max(0, x)$ has a corner at $x = 0$ — the same non-differentiability as $|x|$ shifted. In practice, ML frameworks assign a conventional “derivative” at the kink ($0$ or $1$), and this works because:

1. The set of inputs landing exactly on the kink has Lebesgue measure zero — almost every input avoids it. (Sigma-Algebras & Measures (coming soon))
2. The difference quotient from either side is well-defined (it’s $0$ or $1$), so the “gradient” used in practice is the one-sided derivative — which is fine for optimization.

The Weierstrass function shows that pathology can be extreme — continuous everywhere, differentiable nowhere — but such functions don’t arise as loss landscapes in practice. Real neural network losses are piecewise smooth (smooth between the ReLU kinks), and that’s smooth enough for gradient descent.