The Hessian & Second-Order Analysis

Let $f: \mathbb{R}^n \to \mathbb{R}$ have partial derivative $\frac{\partial f}{\partial x_j}$ in a neighborhood of $a$. If $\frac{\partial f}{\partial x_j}$ is itself differentiable with respect to $x_i$ at $a$, the second-order partial derivative is

$\frac{\partial^2 f}{\partial x_i \partial x_j}(a) = \frac{\partial}{\partial x_i}\left(\frac{\partial f}{\partial x_j}\right)(a).$

When $i = j$, this is the unmixed (or pure) second partial derivative $\frac{\partial^2 f}{\partial x_i^2}$. When $i \neq j$, this is a mixed partial derivative.

Notation: $f_{x_i x_j}(a)$, $\partial_{ij} f(a)$, $D_i D_j f(a)$.

Let $f: \mathbb{R}^n \to \mathbb{R}$ and suppose the mixed partial derivatives $\frac{\partial^2 f}{\partial x_i \partial x_j}$ and $\frac{\partial^2 f}{\partial x_j \partial x_i}$ both exist and are continuous in a neighborhood of $a$. Then

$\frac{\partial^2 f}{\partial x_i \partial x_j}(a) = \frac{\partial^2 f}{\partial x_j \partial x_i}(a).$

In short: if $f \in C^2$ (all second partial derivatives exist and are continuous), the order of differentiation does not matter. This ensures the Hessian matrix is symmetric.

We prove the $n = 2$ case. Let $f: \mathbb{R}^2 \to \mathbb{R}$ with continuous mixed partials $f_{xy}$ and $f_{yx}$ near $(a, b)$. Define the second-difference quotient:

$\Delta(h, k) = f(a+h, b+k) - f(a+h, b) - f(a, b+k) + f(a, b).$

Let $\phi(x) = f(x, b+k) - f(x, b)$. Then $\Delta(h,k) = \phi(a+h) - \phi(a)$. By the Mean Value Theorem, there exists $\xi$ between $a$ and $a+h$ such that:

$\Delta(h,k) = h\,\phi'(\xi) = h\left[f_x(\xi, b+k) - f_x(\xi, b)\right].$

Applying the Mean Value Theorem again to $g(y) = f_x(\xi, y)$, there exists $\eta$ between $b$ and $b+k$ such that:

$\Delta(h,k) = hk \cdot f_{xy}(\xi, \eta).$

By the same argument with the roles of $x$ and $y$ reversed — define $\psi(y) = f(a+h, y) - f(a, y)$ and apply the MVT twice — we obtain:

$\Delta(h,k) = hk \cdot f_{yx}(\xi', \eta')$

for some $\xi'$ between $a$ and $a+h$, $\eta'$ between $b$ and $b+k$.

As $(h, k) \to (0, 0)$, we have $(\xi, \eta) \to (a, b)$ and $(\xi', \eta') \to (a, b)$. Since $f_{xy}$ and $f_{yx}$ are continuous at $(a, b)$:

$f_{xy}(a, b) = \lim_{(h,k) \to (0,0)} \frac{\Delta(h,k)}{hk} = f_{yx}(a, b).$

$\square$

We compute all four second-order partial derivatives:

First partials:

$f_x = 2xy + y\cos(xy), \qquad f_y = x^2 + x\cos(xy).$

Second partials:

$f_{xx} = 2y - y^2\sin(xy), \qquad f_{yy} = -x^2\sin(xy),$

$f_{xy} = 2x + \cos(xy) - xy\sin(xy), \qquad f_{yx} = 2x + \cos(xy) - xy\sin(xy).$

Observe $f_{xy} = f_{yx}$ — Clairaut’s theorem confirmed. Both mixed partials are continuous, so the order of differentiation is interchangeable. The four second partials, assembled into a $2 \times 2$ matrix, form the Hessian.

In single-variable calculus (Topic 5), the second derivative $f''(a)$ is a single number. For $f: \mathbb{R}^n \to \mathbb{R}$, the second-order information consists of $n^2$ second partial derivatives. Clairaut’s theorem reduces this to $\frac{n(n+1)}{2}$ independent values (the upper triangle of a symmetric matrix). For $n = 2$: 3 values ($f_{xx}$, $f_{xy}$, $f_{yy}$). For $n = 100$ (a small neural network): 5,050 values. For $n = 10^8$ (a large language model): storing the full Hessian is infeasible — motivating the approximations discussed in Section 8.

Let $f: \mathbb{R}^n \to \mathbb{R}$ be twice differentiable at $a$. The Hessian matrix of $f$ at $a$ is the $n \times n$ matrix

$H_f(a) = \begin{pmatrix} \frac{\partial^2 f}{\partial x_1^2}(a) & \frac{\partial^2 f}{\partial x_1 \partial x_2}(a) & \cdots & \frac{\partial^2 f}{\partial x_1 \partial x_n}(a) \\[6pt] \frac{\partial^2 f}{\partial x_2 \partial x_1}(a) & \frac{\partial^2 f}{\partial x_2^2}(a) & \cdots & \frac{\partial^2 f}{\partial x_2 \partial x_n}(a) \\[6pt] \vdots & \vdots & \ddots & \vdots \\[6pt] \frac{\partial^2 f}{\partial x_n \partial x_1}(a) & \frac{\partial^2 f}{\partial x_n \partial x_2}(a) & \cdots & \frac{\partial^2 f}{\partial x_n^2}(a) \end{pmatrix}.$

If $f \in C^2$, then $H_f(a)$ is symmetric by Clairaut’s theorem.

Notation: $H_f(a)$, $\nabla^2 f(a)$, $D^2 f(a)$.

Let $f: \mathbb{R}^n \to \mathbb{R}$ be $C^2$. The gradient $\nabla f: \mathbb{R}^n \to \mathbb{R}^n$ is a vector-valued function with component functions $(\nabla f)_i = \frac{\partial f}{\partial x_i}$. Its Jacobian matrix is

$J(\nabla f)(a)_{ij} = \frac{\partial}{\partial x_j}\left(\frac{\partial f}{\partial x_i}\right)(a) = \frac{\partial^2 f}{\partial x_j \partial x_i}(a) = H_f(a)_{ij}.$

Therefore $H_f(a) = J(\nabla f)(a)$. The Hessian is the Jacobian of the gradient. (The last equality uses the symmetry $\frac{\partial^2 f}{\partial x_j \partial x_i} = \frac{\partial^2 f}{\partial x_i \partial x_j}$ guaranteed by Clairaut’s theorem for $C^2$ functions — without symmetry, $J(\nabla f)$ would be the transpose of the Hessian as defined above.)

Direct verification. The $i$-th component of $\nabla f$ is $g_i(x) = \frac{\partial f}{\partial x_i}(x)$. By the definition of the Jacobian (Topic 10), the Jacobian of $g = \nabla f$ has entries:

$J_g(a)_{ij} = \frac{\partial g_i}{\partial x_j}(a) = \frac{\partial}{\partial x_j}\frac{\partial f}{\partial x_i}(a) = \frac{\partial^2 f}{\partial x_j \partial x_i}(a).$

By Clairaut’s theorem (Theorem 1), $\frac{\partial^2 f}{\partial x_j \partial x_i} = \frac{\partial^2 f}{\partial x_i \partial x_j} = H_f(a)_{ij}$. Therefore $J(\nabla f)(a) = H_f(a)$.

$\square$

$f(x,y) = x^2 + 4y^2$.

Gradient: $\nabla f = (2x, 8y)$.

Hessian:

$H_f = \begin{pmatrix} 2 & 0 \\ 0 & 8 \end{pmatrix}$

— constant, independent of $(x,y)$. The eigenvalues are 2 and 8. The surface curves 4 times more steeply in the $y$-direction than in the $x$-direction. This is the prototypical ill-conditioned loss surface: gradient descent zigzags because the curvatures are unequal. The condition number is $\kappa = 8/2 = 4$.

$f(x,y) = x^2 - y^2$.

Gradient: $\nabla f = (2x, -2y)$.

Hessian:

$H_f = \begin{pmatrix} 2 & 0 \\ 0 & -2 \end{pmatrix}.$

One positive eigenvalue (2, curves upward in $x$) and one negative eigenvalue ($-2$, curves downward in $y$). The origin is a saddle point — a critical point that is neither a minimum nor a maximum. The surface looks like a horse saddle: it rises in one direction and descends in the other.

Consider a loss $L(\theta_1, \theta_2) = (y - \sigma(\theta_1 x_1 + \theta_2 x_2))^2$ where $\sigma$ is the sigmoid function. The Hessian $H_L(\theta)$ has entries involving $\sigma'$ and $\sigma''$ — it depends on the data $(x_1, x_2, y)$ and the current parameters $\theta$. Unlike the paraboloid, this Hessian is not constant — the curvature of the loss surface changes as the parameters move. Computing this $2 \times 2$ matrix is cheap; computing a $10^8 \times 10^8$ matrix for a real network is not. This is why practical deep learning relies on Hessian approximations rather than the full matrix.

A symmetric $n \times n$ matrix $A$ is:

• Positive definite ($A \succ 0$) if $h^T A h > 0$ for all $h \neq 0$. Equivalently, all eigenvalues of $A$ are positive.
• Negative definite ($A \prec 0$) if $h^T A h < 0$ for all $h \neq 0$. Equivalently, all eigenvalues are negative.
• Positive semidefinite ($A \succeq 0$) if $h^T A h \ge 0$ for all $h$. Equivalently, all eigenvalues are nonnegative.
• Negative semidefinite ($A \preceq 0$) if $h^T A h \le 0$ for all $h$. Equivalently, all eigenvalues are nonpositive.
• Indefinite if $h^T A h$ takes both positive and negative values. Equivalently, $A$ has both positive and negative eigenvalues.

Let $A$ be a symmetric $n \times n$ matrix with eigenvalues $\lambda_1 \le \lambda_2 \le \cdots \le \lambda_n$. Then:

(a) $A \succ 0 \iff \lambda_1 > 0$.

(b) $A \prec 0 \iff \lambda_n < 0$.

(c) $A$ is indefinite $\iff \lambda_1 < 0 < \lambda_n$.

For $n = 2$: $A = \begin{pmatrix} a & b \\ b & c \end{pmatrix}$ is positive definite iff $a > 0$ and $\det A = ac - b^2 > 0$. It is indefinite iff $\det A < 0$.

A point $a$ is a saddle point of $f$ if $\nabla f(a) = 0$ and $H_f(a)$ is indefinite — i.e., $f$ curves upward in some directions and downward in others. Near a saddle point, $f(a + h) > f(a)$ for some directions $h$ and $f(a + h) < f(a)$ for others. The origin of $f(x,y) = x^2 - y^2$ is the canonical example.

Let $f: \mathbb{R}^n \to \mathbb{R}$ be $C^2$ near $a$, and suppose $\nabla f(a) = 0$ (so $a$ is a critical point). Then:

1. If $H_f(a) \succ 0$ (positive definite), then $a$ is a strict local minimum.
2. If $H_f(a) \prec 0$ (negative definite), then $a$ is a strict local maximum.
3. If $H_f(a)$ is indefinite, then $a$ is a saddle point.
4. If $H_f(a)$ is positive or negative semidefinite (has a zero eigenvalue), the test is inconclusive — higher-order analysis is needed.

For $n = 2$, write $H_f = \begin{pmatrix} f_{xx} & f_{xy} \\ f_{xy} & f_{yy} \end{pmatrix}$ and let $D = f_{xx} f_{yy} - f_{xy}^2 = \det H_f$. Then:

• $D > 0$ and $f_{xx} > 0$: local minimum.
• $D > 0$ and $f_{xx} < 0$: local maximum.
• $D < 0$: saddle point.
• $D = 0$: inconclusive.

We sketch the proof via the second-order Taylor expansion (proven in Section 5). At a critical point where $\nabla f(a) = 0$:

$f(a+h) = f(a) + \frac{1}{2} h^T H_f(a) h + o(\|h\|^2).$

Case 1: $H_f(a) \succ 0$. By the spectral theorem for symmetric matrices, $h^T H_f(a) h \ge \lambda_{\min} \|h\|^2$ where $\lambda_{\min} > 0$ is the smallest eigenvalue. For $\|h\|$ sufficiently small, the $o(\|h\|^2)$ error is dominated by $\frac{1}{2}\lambda_{\min}\|h\|^2$, so $f(a+h) > f(a)$ for all $h \neq 0$ in a neighborhood — $a$ is a strict local minimum.

Case 2: $H_f(a) \prec 0$. Analogous, with all inequalities reversed.

Case 3: $H_f(a)$ indefinite. Let $v_+$ be an eigenvector with positive eigenvalue $\lambda_+ > 0$ and $v_-$ an eigenvector with negative eigenvalue $\lambda_- < 0$. Along $h = tv_+$ for small $t > 0$: $f(a + tv_+) \approx f(a) + \frac{1}{2}\lambda_+ t^2 > f(a)$. Along $h = tv_-$: $f(a + tv_-) \approx f(a) + \frac{1}{2}\lambda_- t^2 < f(a)$. So $a$ is neither a local min nor a local max — it is a saddle point.

$\square$

Gradient: $\nabla f = (4x^3 - 4x,\; 4y^3 - 4y)$. Setting both components to zero: $4x(x^2 - 1) = 0$ and $4y(y^2 - 1) = 0$. The critical points are all combinations of $x \in \{-1, 0, 1\}$ and $y \in \{-1, 0, 1\}$ — nine points total.

Hessian:

$H_f = \begin{pmatrix} 12x^2 - 4 & 0 \\ 0 & 12y^2 - 4 \end{pmatrix}.$

At $(0,0)$: $H_f = \begin{pmatrix} -4 & 0 \\ 0 & -4 \end{pmatrix} \prec 0$ — local maximum.

At $(1,0)$: $H_f = \begin{pmatrix} 8 & 0 \\ 0 & -4 \end{pmatrix}$ — indefinite — saddle point. Similarly for $(-1,0)$, $(0,1)$, $(0,-1)$.

At $(1,1)$: $H_f = \begin{pmatrix} 8 & 0 \\ 0 & 8 \end{pmatrix} \succ 0$ — local minimum. Similarly for $(-1,1)$, $(1,-1)$, $(-1,-1)$.

Nine critical points: 1 local max, 4 saddle points, 4 local minima.

The function $f(x,y) = x^4 + y^4$ has $\nabla f(0,0) = 0$ and $H_f(0,0) = \begin{pmatrix} 0 & 0 \\ 0 & 0 \end{pmatrix}$ — the zero matrix. The test is inconclusive. Yet the origin is clearly a local (and global) minimum: $f(x,y) = x^4 + y^4 > 0 = f(0,0)$ for all $(x,y) \neq (0,0)$.

The issue is that the second-order Taylor expansion is not informative when the quadratic term vanishes — the fourth-order terms dominate. Compare with $g(x,y) = x^4 - y^4$: same zero Hessian at the origin, but now the origin is a saddle point (of fourth order). The zero Hessian tells us the second-order analysis has nothing to say — we need higher-order derivatives to classify the point.

Let $f: \mathbb{R}^n \to \mathbb{R}$ be $C^2$ in a neighborhood of $a$. Then for all $h$ with $a + h$ in that neighborhood:

$f(a + h) = f(a) + \nabla f(a) \cdot h + \frac{1}{2} h^T H_f(a)\, h + R_2(h)$

where the remainder satisfies $\lim_{h \to 0} \frac{R_2(h)}{\|h\|^2} = 0$, i.e., $R_2(h) = o(\|h\|^2)$.

In expanded form:

$f(a+h) = f(a) + \sum_{i=1}^n \frac{\partial f}{\partial x_i}(a)\, h_i + \frac{1}{2} \sum_{i=1}^n \sum_{j=1}^n \frac{\partial^2 f}{\partial x_i \partial x_j}(a)\, h_i h_j + o(\|h\|^2).$

This is the multivariable generalization of $f(a+h) = f(a) + f'(a)h + \frac{1}{2}f''(a)h^2 + o(h^2)$ from Topic 6.

Apply the single-variable Taylor expansion to the function $g(t) = f(a + th)$ at $t = 0$:

$g(t) = g(0) + g'(0)\,t + \frac{1}{2}g''(0)\,t^2 + o(t^2).$

We compute the derivatives of $g$:

• $g(0) = f(a)$.
• $g'(t) = \nabla f(a + th) \cdot h$ by the chain rule. So $g'(0) = \nabla f(a) \cdot h$.
• $g''(t) = \sum_{i,j} \frac{\partial^2 f}{\partial x_i \partial x_j}(a + th)\, h_i h_j = h^T H_f(a + th)\, h$. So $g''(0) = h^T H_f(a)\, h$.

Setting $t = 1$:

$f(a+h) = g(1) = f(a) + \nabla f(a) \cdot h + \frac{1}{2} h^T H_f(a)\, h + o(\|h\|^2).$

The error bound conversion from $o(t^2)$ at $t = 1$ to $o(\|h\|^2)$ uses the continuity of the second partial derivatives.

$\square$

At $(0, 0)$: $f(0,0) = 1$, $\nabla f(0,0) = (1, 1)$, $H_f(0,0) = \begin{pmatrix} 1 & 1 \\ 1 & 1 \end{pmatrix}$.

The second-order Taylor expansion:

$e^{x+y} \approx 1 + (x + y) + \frac{1}{2}(x^2 + 2xy + y^2) = 1 + (x+y) + \frac{1}{2}(x+y)^2.$

This matches the known Taylor series $e^u \approx 1 + u + \frac{1}{2}u^2$ with $u = x + y$. The Hessian captures the curvature of $e^{x+y}$ at the origin: it curves equally in all directions within the subspace $\{(t, t) : t \in \mathbb{R}\}$ and is flat in the perpendicular direction — reflected by the eigenvalues 0 and 2.

For $f(x,y) = ax^2 + by^2$ with $a, b > 0$:

$H_f = \begin{pmatrix} 2a & 0 \\ 0 & 2b \end{pmatrix}.$

Eigenvalues: $\lambda_1 = 2a$, $\lambda_2 = 2b$, with eigenvectors along the coordinate axes. The condition number $\kappa = \lambda_{\max}/\lambda_{\min} = \max(a,b)/\min(a,b)$ measures eccentricity.

When $a = b$: $\kappa = 1$ (circular contours, isotropic curvature — gradient descent converges in a straight line).

When $a \gg b$ or $a \ll b$: $\kappa \gg 1$ (elliptical contours, anisotropic curvature — gradient descent zigzags). The severity of the zigzag is directly proportional to $\kappa$.

Gradient: $\nabla f = (3x^2 - 3y^2,\; -6xy)$, so $(0,0)$ is a critical point.

Hessian at origin:

$H_f(0,0) = \begin{pmatrix} 6x & -6y \\ -6y & -6x \end{pmatrix}\bigg|_{(0,0)} = \begin{pmatrix} 0 & 0 \\ 0 & 0 \end{pmatrix}.$

The second derivative test is inconclusive. Yet the surface has three “valleys” meeting at the origin (picture a monkey sitting with both legs and a tail hanging down). This is a degenerate critical point requiring third-order analysis — the Hessian’s silence here tells us that the curvature is zero in all directions, so the second-order Taylor expansion provides no useful local information.

For a quadratic $f(x) = \frac{1}{2}x^T A x - b^T x + c$ with $A \succ 0$, gradient descent with optimal step size converges in $O(\kappa \log(1/\epsilon))$ iterations, where $\kappa = \lambda_{\max}(A)/\lambda_{\min}(A)$ is the condition number. Newton’s method converges in $O(\log\log(1/\epsilon))$ iterations (quadratic convergence).

The gap is dramatic: for $\kappa = 1000$, gradient descent needs roughly 7,000 steps while Newton needs roughly 10. The condition number $\kappa(H_f)$ at a local minimum of a non-quadratic loss measures the local version of this problem — it tells you how much the loss landscape stretches gradient descent steps in the worst direction relative to the best.

At the current iterate $x_k$, approximate $f$ by its second-order Taylor expansion:

$f(x_k + h) \approx f(x_k) + \nabla f(x_k) \cdot h + \frac{1}{2} h^T H_f(x_k)\, h.$

This is a quadratic function of $h$. Its gradient with respect to $h$ is $\nabla f(x_k) + H_f(x_k)\, h$, which vanishes when:

$h^* = -H_f(x_k)^{-1}\, \nabla f(x_k).$

The Newton update is:

$x_{k+1} = x_k + h^* = x_k - H_f(x_k)^{-1}\, \nabla f(x_k).$

Compare with gradient descent: $x_{k+1} = x_k - \eta\, \nabla f(x_k)$. Newton replaces the scalar step size $\eta$ with the matrix $H_f^{-1}$ — a direction- and curvature-dependent step that accounts for the local shape of the surface.

$f(x,y) = (1-x)^2 + 100(y-x^2)^2$.

The minimum is at $(1, 1)$. Gradient descent converges slowly because the loss landscape is a narrow, curved valley with condition number $\kappa \approx 2500$ at the minimum. The gradient points nearly perpendicular to the valley floor, so GD bounces between the walls with each step.

Newton’s method converges in a few iterations because it uses the Hessian to navigate the curvature. The Hessian-inverse premultiplication rotates the gradient to align with the valley floor and scales the step to match the local curvature — exactly the correction that gradient descent is missing.

Newton’s method requires $H_f(x_k)$ to be positive definite — otherwise the quadratic model has no minimum (it has a maximum or saddle). At a saddle point, $H_f$ is indefinite, and the Newton step may move toward the saddle rather than away from it.

This is why second-order methods in practice often use modified Newton methods that ensure the step direction is always a descent direction. Common modifications include:

• Levenberg-Marquardt damping: Replace $H_f^{-1}$ with $(H_f + \mu I)^{-1}$ for some $\mu > 0$, shifting all eigenvalues to be positive.
• Trust region methods: Constrain $\|h\| \le \Delta$ and solve the constrained quadratic subproblem.
• Eigenvalue modification: Replace negative eigenvalues with their absolute values before inverting.