Learning goal: on suitable elongated quadratics, momentum tuning depends strongly on β and the condition number κ. The displayed β* assumes its matching optimal quadratic step size. Adam is approximately invariant to positive global gradient rescaling when ε is negligible. Multiplying the loss by 10³ changes fixed-step GD and momentum, while Adam's trajectory can remain nearly unchanged. But shifting a feature by a large constant can create severe conditioning; Adam does not automatically remove that problem.

Contours: loss; paths: 300 steps of GD, momentum, and Adam; cross: optimum.

Log excess loss versus steps, converted back to the unscaled objective for comparison.

Observe: tuning β toward the displayed quadratic reference can improve momentum's curve here. Excessive momentum can overshoot. Scaling the loss by 10³ destabilizes the fixed GD and momentum steps, while Adam changes little in this example because its gradient-moment ratio largely cancels a global scale. The +1911 feature offset instead gives extreme conditioning, and all three fixed training setups perform poorly. Standardization improves this comparison. Update rules and feature geometry are separate interventions; Adam's second moment is not an exact Hessian correction.