Partial Derivatives & the Gradient

Let $f: \mathbb{R}^n \to \mathbb{R}$ and let $a = (a_1, \ldots, a_n) \in \mathbb{R}^n$. The partial derivative of $f$ with respect to $x_i$ at $a$ is

$\frac{\partial f}{\partial x_i}(a) = \lim_{h \to 0} \frac{f(a_1, \ldots, a_i + h, \ldots, a_n) - f(a)}{h},$

provided this limit exists. In plain English: hold every input fixed except $x_i$, and take the ordinary derivative with respect to $x_i$.

Notation: $\frac{\partial f}{\partial x_i}(a) = f_{x_i}(a) = \partial_i f(a) = D_i f(a)$.

To compute $\frac{\partial f}{\partial x}$: treat $y$ as a constant and differentiate with respect to $x$.

$\frac{\partial f}{\partial x} = 2xy.$

To compute $\frac{\partial f}{\partial y}$: treat $x$ as a constant and differentiate with respect to $y$.

$\frac{\partial f}{\partial y} = x^2 + \cos(y).$

At the point $(1, \pi/2)$: $f_x(1, \pi/2) = 2 \cdot 1 \cdot \pi/2 = \pi$ and $f_y(1, \pi/2) = 1 + \cos(\pi/2) = 1$.

The chain rule from Topic 5 applies directly. Treating $y$ as constant:

$\frac{\partial f}{\partial x} = 2x \, e^{x^2 + y^2}.$

Treating $x$ as constant:

$\frac{\partial f}{\partial y} = 2y \, e^{x^2 + y^2}.$

At any point, $f_x$ and $f_y$ have the same exponential factor — they differ only in the leading coefficient ($2x$ vs. $2y$). This symmetry reflects the rotational structure of $x^2 + y^2$.

If $f: \mathbb{R}^2 \to \mathbb{R}$ is differentiable at $a = (a_1, a_2)$, then

$f(a_1 + h_1, a_2 + h_2) \approx f(a) + f_x(a) \, h_1 + f_y(a) \, h_2,$

with error that vanishes faster than $\|(h_1, h_2)\|$ as $(h_1, h_2) \to (0, 0)$. The surface $z = f(a) + f_x(a)(x - a_1) + f_y(a)(y - a_2)$ is the tangent plane to $z = f(x,y)$ at $(a_1, a_2, f(a))$.

In $\mathbb{R}^1$, the best linear approximation is a line. In $\mathbb{R}^2$, it is a plane. In $\mathbb{R}^n$, it is a hyperplane:

$f(a + h) \approx f(a) + \sum_{i=1}^n \frac{\partial f}{\partial x_i}(a) \, h_i.$

The pattern is the same — the derivative provides the coefficients of the linear approximation. In the next topic (The Jacobian & Multivariate Chain Rule), the derivative of a vector-valued function becomes a matrix, and the tangent hyperplane becomes an affine subspace.

Let $f: \mathbb{R}^n \to \mathbb{R}$ be a function whose partial derivatives all exist at $a$. The gradient of $f$ at $a$ is the vector

$\nabla f(a) = \left(\frac{\partial f}{\partial x_1}(a), \frac{\partial f}{\partial x_2}(a), \ldots, \frac{\partial f}{\partial x_n}(a)\right) \in \mathbb{R}^n.$

Notation: $\nabla f(a) = \text{grad}\, f(a) = Df(a)^T$. We follow the column-vector convention throughout: $\nabla f$ is a column vector in $\mathbb{R}^n$.

$\nabla f(x, y) = (2x, 2y).$

At any point, the gradient points radially outward from the origin — directly “uphill” on the paraboloid. At $(1, 1)$: $\nabla f = (2, 2)$, pointing toward the northeast. The magnitude $\|\nabla f(1,1)\| = \sqrt{8} = 2\sqrt{2}$ measures how steep the surface is at that point.

$\nabla f = (yz, \, xz, \, xy).$

At $(1, 2, 3)$: $\nabla f = (6, 3, 2)$. The function is most sensitive to changes in $x$ (rate $= 6$), less sensitive to $y$ (rate $= 3$), and least sensitive to $z$ (rate $= 2$). In an ML context, this is feature importance: the gradient magnitude per coordinate tells you which inputs matter most.

Let $f: \mathbb{R}^n \to \mathbb{R}$, $a \in \mathbb{R}^n$, and $\mathbf{u} \in \mathbb{R}^n$ a unit vector ($\|\mathbf{u}\| = 1$). The directional derivative of $f$ at $a$ in the direction $\mathbf{u}$ is

$D_\mathbf{u}f(a) = \lim_{t \to 0} \frac{f(a + t\mathbf{u}) - f(a)}{t},$

provided this limit exists. In plain English: stand at $a$, walk in direction $\mathbf{u}$, and measure how fast $f$ changes.

If $f: \mathbb{R}^n \to \mathbb{R}$ is differentiable at $a$, then for every unit vector $\mathbf{u}$, the directional derivative exists and equals

$D_\mathbf{u}f(a) = \nabla f(a) \cdot \mathbf{u}.$

Define $\varphi(t) = f(a + t\mathbf{u})$. This is a function $\varphi: \mathbb{R} \to \mathbb{R}$ — a single-variable function obtained by restricting $f$ to the line through $a$ in direction $\mathbf{u}$. By definition, $D_\mathbf{u}f(a) = \varphi'(0)$.

Since $f$ is differentiable at $a$, the chain rule (The Derivative & Chain Rule, Theorem 6) applies to the composition $\varphi(t) = f(a + t\mathbf{u})$. The “inner function” is $\gamma(t) = a + t\mathbf{u}$, with $\gamma'(0) = \mathbf{u}$. The chain rule gives:

$\varphi'(0) = \sum_{i=1}^n \frac{\partial f}{\partial x_i}(a) \cdot u_i = \nabla f(a) \cdot \mathbf{u}.$

The chain rule is valid here because $f$ is differentiable at $a$ — not merely having partial derivatives. This distinction is critical and we return to it in Section 7. ∎

At $(1, 1)$: $\nabla f = (2, 2)$. In the direction $\mathbf{u} = \frac{1}{\sqrt{2}}(1, 1)$:

$D_\mathbf{u}f(1, 1) = (2, 2) \cdot \frac{1}{\sqrt{2}}(1, 1) = \frac{2 + 2}{\sqrt{2}} = 2\sqrt{2} \approx 2.83.$

Compare this with the partial derivatives: $f_x(1,1) = 2$ and $f_y(1,1) = 2$. The directional derivative in the diagonal direction is larger than either partial derivative individually — because the gradient points diagonally, and we are measuring rate of change in the gradient’s own direction.

Let $f: \mathbb{R}^n \to \mathbb{R}$ be differentiable at $a$ with $\nabla f(a) \neq 0$. Then among all unit vectors $\mathbf{u} \in \mathbb{R}^n$:

1. The maximum of $D_\mathbf{u}f(a)$ is $\|\nabla f(a)\|$, achieved when $\mathbf{u} = \frac{\nabla f(a)}{\|\nabla f(a)\|}$.
2. The minimum of $D_\mathbf{u}f(a)$ is $-\|\nabla f(a)\|$, achieved when $\mathbf{u} = -\frac{\nabla f(a)}{\|\nabla f(a)\|}$.
3. $D_\mathbf{u}f(a) = 0$ if and only if $\mathbf{u}$ is orthogonal to $\nabla f(a)$.

By Theorem 1, $D_\mathbf{u}f = \nabla f \cdot \mathbf{u}$. By the Cauchy-Schwarz inequality:

$|\nabla f \cdot \mathbf{u}| \le \|\nabla f\| \cdot \|\mathbf{u}\| = \|\nabla f\|,$

since $\|\mathbf{u}\| = 1$. Equality holds if and only if $\mathbf{u}$ is a scalar multiple of $\nabla f$. Since $\|\mathbf{u}\| = 1$, this means $\mathbf{u} = \pm \nabla f / \|\nabla f\|$.

The positive sign gives $\nabla f \cdot (\nabla f / \|\nabla f\|) = \|\nabla f\|^2 / \|\nabla f\| = \|\nabla f\|$ (the maximum). The negative sign gives $-\|\nabla f\|$ (the minimum). If $\mathbf{u} \perp \nabla f$, then $\nabla f \cdot \mathbf{u} = 0$. ∎

Let $f: \mathbb{R}^n \to \mathbb{R}$ be differentiable at $a$ with $\nabla f(a) \neq 0$. Let $S = \{x \in \mathbb{R}^n : f(x) = f(a)\}$ be the level set through $a$. If $\gamma: (-\varepsilon, \varepsilon) \to \mathbb{R}^n$ is a smooth curve with $\gamma(0) = a$ and $\gamma(t) \in S$ for all $t$, then

$\nabla f(a) \cdot \gamma'(0) = 0.$

In words: the gradient is perpendicular to every curve that stays on the level set.

Since $f(\gamma(t)) = f(a) = c$ for all $t$ (the curve stays on the level set), differentiating with respect to $t$ at $t = 0$ gives:

$\frac{d}{dt} f(\gamma(t))\bigg|_{t=0} = \nabla f(\gamma(0)) \cdot \gamma'(0) = \nabla f(a) \cdot \gamma'(0) = 0,$

by the chain rule. This holds for every tangent vector $\gamma'(0)$ to $S$ at $a$, so $\nabla f(a)$ is orthogonal to the tangent space of $S$ at $a$. ∎

On a topographic map, contour lines connect points at the same elevation. The gradient points in the direction of steepest climb — straight up the hillside, perpendicular to the contour lines. A hiker following the gradient path reaches the summit as fast as possible; a hiker walking along a contour line (perpendicular to the gradient) stays at the same elevation. Gradient descent reverses this: it follows $-\nabla f$, going downhill as steeply as possible.

For a loss function with circular contours (isotropic loss, like $L = x^2 + y^2$), the gradient points radially inward and gradient descent takes a straight path to the minimum. For elongated contours (anisotropic loss, like $L = x^2 + 10y^2$), the gradient does not point toward the minimum — it points perpendicular to the contour, which may oscillate back and forth across the narrow valley. This is exactly why momentum and adaptive methods like Adam were invented.

A function $f: \mathbb{R}^n \to \mathbb{R}$ is differentiable at $a$ if there exists a linear map $Df(a): \mathbb{R}^n \to \mathbb{R}$ such that

$\lim_{h \to 0} \frac{|f(a+h) - f(a) - Df(a)(h)|}{\|h\|} = 0.$

When $f$ is differentiable, $Df(a)$ is unique and is represented by the gradient: $Df(a)(h) = \nabla f(a) \cdot h$. For $f: \mathbb{R}^n \to \mathbb{R}^m$, $Df(a)$ is represented by the Jacobian matrix — that is the next topic.

If $f: \mathbb{R}^n \to \mathbb{R}$ is differentiable at $a$, then all partial derivatives exist at $a$, and $Df(a)(\mathbf{e}_i) = \frac{\partial f}{\partial x_i}(a)$.

Set $h = t\mathbf{e}_i$ (approach along the $i$-th coordinate axis) in the total derivative definition:

$\frac{|f(a + t\mathbf{e}_i) - f(a) - Df(a)(t\mathbf{e}_i)|}{|t|} \to 0 \quad \text{as } t \to 0.$

Since $Df(a)$ is linear, $Df(a)(t\mathbf{e}_i) = t \cdot Df(a)(\mathbf{e}_i)$. Rearranging:

$\left|\frac{f(a + t\mathbf{e}_i) - f(a)}{t} - Df(a)(\mathbf{e}_i)\right| \to 0.$

The left side of the subtraction is exactly the difference quotient defining $\frac{\partial f}{\partial x_i}(a)$. So the partial derivative exists and equals $Df(a)(\mathbf{e}_i)$. ∎

Partial derivatives existing does not imply differentiability. This echoes the single-variable situation (continuity does not imply differentiability), but in a stronger form: in $\mathbb{R}^n$, we can have all partial derivatives at a point and still fail to be continuous there. The following example demonstrates this dramatically.

Define

$f(x, y) = \begin{cases} \frac{xy}{x^2 + y^2} & (x, y) \neq (0, 0) \\ 0 & (x, y) = (0, 0). \end{cases}$

Partial derivatives at the origin exist and equal zero:

$f_x(0, 0) = \lim_{h \to 0} \frac{f(h, 0) - f(0, 0)}{h} = \lim_{h \to 0} \frac{0}{h} = 0.$

Similarly $f_y(0, 0) = 0$. Both partial derivatives exist because along the coordinate axes, $f$ is identically zero.

But $f$ is not continuous at the origin: Along the line $y = x$,

$f(x, x) = \frac{x \cdot x}{x^2 + x^2} = \frac{x^2}{2x^2} = \frac{1}{2} \neq 0 = f(0, 0).$

The function approaches $1/2$ along the diagonal but equals $0$ at the origin. Since $f$ is not continuous, it certainly cannot be differentiable — yet both partial derivatives exist. The moral: partial derivatives probe only the coordinate directions; differentiability requires consistent behavior from all directions simultaneously.

If all partial derivatives $\frac{\partial f}{\partial x_i}$ exist in a neighborhood of $a$ and are continuous at $a$, then $f$ is differentiable at $a$.

We sketch the proof for $n = 2$; the general case is analogous. Write:

$f(a + h) - f(a) = [f(a_1 + h_1, a_2 + h_2) - f(a_1, a_2 + h_2)] + [f(a_1, a_2 + h_2) - f(a_1, a_2)].$

Apply the single-variable Mean Value Theorem (Mean Value Theorem & Taylor Expansion, Theorem 1) to each bracket:

• The first bracket equals $f_x(c_1, a_2 + h_2) \cdot h_1$ for some $c_1$ between $a_1$ and $a_1 + h_1$.
• The second bracket equals $f_y(a_1, c_2) \cdot h_2$ for some $c_2$ between $a_2$ and $a_2 + h_2$.

So:

$f(a + h) - f(a) = f_x(c_1, a_2 + h_2) \cdot h_1 + f_y(a_1, c_2) \cdot h_2.$

Now subtract the linear approximation $f_x(a) h_1 + f_y(a) h_2$:

$f(a+h) - f(a) - [f_x(a) h_1 + f_y(a) h_2] = [f_x(c_1, a_2+h_2) - f_x(a)] h_1 + [f_y(a_1, c_2) - f_y(a)] h_2.$

By continuity of $f_x$ and $f_y$ at $a$: as $h \to 0$, we have $c_1 \to a_1$ and $c_2 \to a_2$, so $f_x(c_1, a_2 + h_2) \to f_x(a)$ and $f_y(a_1, c_2) \to f_y(a)$. Call these differences $\alpha(h)$ and $\beta(h)$, both tending to zero. Then:

$|f(a+h) - f(a) - \nabla f(a) \cdot h| \le |\alpha(h)| \cdot |h_1| + |\beta(h)| \cdot |h_2| \le (|\alpha(h)| + |\beta(h)|) \|h\|.$

Dividing by $\|h\|$: the error ratio is at most $|\alpha(h)| + |\beta(h)| \to 0$. ∎

Just as in The Derivative & Chain Rule, differentiability at $a$ implies continuity at $a$:

$|f(a + h) - f(a)| = |Df(a)(h) + o(\|h\|)| \le \|Df(a)\| \cdot \|h\| + o(\|h\|) \to 0.$

The chain of implications is:

$C^1 \;\Rightarrow\; \text{differentiable} \;\Rightarrow\; \text{continuous}, \qquad \text{differentiable} \;\Rightarrow\; \text{partial derivatives exist},$

but none of the converses hold in general. The $C^1$ criterion is the practical workhorse: in virtually every function arising in machine learning (polynomials, exponentials, compositions of smooth functions), the partial derivatives are continuous, so differentiability is automatic.