Gradient descent: steep directions limit steps, flat directions limit progress

Learning goal: on a positive-definite quadratic, the stability boundary is η* = 2/λ _max. The convergence rate depends on the condition number κ = λ _max/λ_min. Rescaling features changes the Hessian and stability scale. Standardization can improve conditioning, but need not make the bowl perfectly round.

Contours: loss; polyline: gradient-descent path from a fixed start; cross: optimum.

Log loss versus steps. Dashed: theoretical contraction at half the stability threshold.

η relative to η* Feature scale s

Standardize columns

λmax = — κ = — η* = — Current η = — Loss after 50 steps = —

Observe: beyond η*, a nonzero unstable component changes from decaying oscillation to growing oscillation. The log-loss curve rises. Feature scaling changes λ _max and η*, sometimes by orders of magnitude. Standardization can reduce κ and accelerate this comparison. It changes the optimization geometry, rather than being only a formatting step.