M0.0 Descending Through Fog: Gradients and Direction

Fog hides the mountain

You stand on a mountainside in dense fog. Your phone has no signal, and you can only feel the ground’s local tilt beneath your feet.

A natural strategy is to step toward the steepest descent, then check the slope again.

How can a few local readings identify the steepest of infinitely many directions?

First solve this local question. Whether it reaches the lowest valley, and how far one may step, are separate questions.

DiscussionQ1

What function describes this situation, what information do we have, and what quantity are we seeking?

Show the reasoning

The mountain is the graph of a height function $h(x,y)$ , mapping ground position to elevation.

We do not know the whole function. We only observe how height changes under small movements in different directions at the current position.

We seek the direction with the most negative local slope. A gradient organizes these directional readings into one object on which we can optimize.

Think of a tilted tabletop

A tilted planar table has infinitely many directions, but its tilt has a single downhill orientation and steepness. A marble initially accelerates downhill.

Locally approximating the ground by a plane makes every directional slope depend on that same tilt. The steepest direction, a perpendicular level direction, and intermediate slopes follow a cosine relationship.

Two perpendicular slope readings determine the plane’s tilt. We do not need to test every direction independently.

Two coordinate slopes determine a gradient and every directional derivative of a local plane.

Figure: the gradient points uphill; perpendicular directions have zero first-order height change.

Express the tilt mathematically

Directional derivative. At $p=(x,y)$ , move distance $s$ along a unit vector $u$ . Define

D_u h(p)=\lim_{s\to0}\frac{h(p+su)-h(p)}{s}.

The coordinate directions give the familiar partial derivatives.

Gradient. Assemble those derivatives into a vector:

\nabla h(p)=\Big(\frac{\partial h}{\partial x}(p),\ \frac{\partial h}{\partial y}(p)\Big).

Differentiability means local planar approximation, which implies

D_u h(p)=\nabla h(p)\cdot u .

The leading height change is $s\nabla h(p)\cdot u$ . Dividing by distance gives the directional slope.

DetailsDerive the directional derivative from differentiability

Differentiability gives $h(p+v)=h(p)+g\cdot v+o(\|v\|)$ . Substitute $v=su$ , divide by $s$ , and take the limit to obtain $D_uh=g\cdot u$ . Coordinate directions show that the components of $g$ are the partial derivatives, so $g=\nabla h$ .

Steepest direction. Minimize $\nabla h\cdot u$ over unit vectors. Cauchy–Schwarz bounds its magnitude by $\|\nabla h\|$ , with the minimum attained in the opposite direction:

\min_{\|u\|=1}D_uh=-\|\nabla h\|, \qquad u=-\frac{\nabla h}{\|\nabla h\|}\quad(\nabla h\ne0).

Equivalently, $D_uh=\|\nabla h\|\cos\theta$ . Perpendicular directions are level only to first order, not necessarily along a long straight movement. At zero gradient, first-order information provides no preferred direction.

Why use a log gradient?

For positive quantities such as signal intensity or population density, relative change can matter more than absolute change.

For $f>0$ , the chain rule gives

\nabla\log f=\frac{\nabla f}{f}.

This is absolute change divided by current value. Multiplying $f$ by a positive constant leaves its log gradient unchanged, so an unknown normalization does not affect it.

For the bell-shaped function $f(x)=\exp(-\|x-\mu\|^2/(2\sigma^2))$ ,

\nabla\log f(x)=-\frac{x-\mu}{\sigma^2}.

The vector points toward the center, grows with distance, and is stronger for a narrower bell. It is a local density-increase direction.

Check three intuitive claims

Must we test every direction? No, for differentiable terrain the coordinate derivatives determine all directional slopes. At a crease such as $|x|$ , there may be no gradient or single local plane, so this argument fails.

Does local steepest descent find the lowest valley? Not generally. A stationary point can be a minimum, maximum, or saddle. For differentiable convex functions, stationary points are global minimizers, but need not be unique; convergence still requires suitable assumptions and steps.

Can a steeper slope justify an arbitrarily longer step? No. The approximation is local. Rapidly changing slopes can make a large step overshoot a valley. M0.2 quantifies this limitation.

Try the descent

This bowl-shaped example compares stable and excessive step sizes:

import numpy as np
grad = lambda p: np.array([2*p[0], 4*p[1]])
def descend(eta, steps=30, p=np.array([2.0, 1.5])):
    hs = []
    for _ in range(steps):
        p = p - eta * grad(p)
        hs.append(p[0]**2 + 2*p[1]**2)
    return np.array(hs)
print(descend(0.1)[-1])  # stable: height approaches zero
print(descend(0.6)[-1])  # y multiplier is -1.4: divergence

Check your understanding

References

Strang, G. Calculus. Wellesley-Cambridge Press; MIT OpenCourseWare.
Marsden, J. E., Tromba, A. J. Vector Calculus. W. H. Freeman.
Nocedal, J., Wright, S. J. Numerical Optimization, 2nd ed. Springer 2006.