SGD noise floors and the small-step η/B approximation

Learning goal: constant-step SGD with persistent noise can fluctuate around an optimum at a noise floor. In a small-step approximation, the ratio η/B controls its scale, with curvature, sampling, and state-dependent noise also contributing.

Contours: full-data loss; path: last 300 SGD steps; cloud: last 1500 iterates; cross: optimum.

Log full-data excess loss versus steps; dashed: median over the final 1500 iterates.

η relative to η* Batch B

Sample with replacement Decay η over steps

η/B = — Observed floor = — Floor ÷ (η/B) = — Other pair's floor = — Other ratio = — κ = —

Observe: overlay a second setting whose η and B are both four times larger. In the small-step regime, the noise-floor scales can be similar. Without replacement, a large batch can give a lower floor because of the finite-population variance factor (n−B)/(n−1). A full batch removes minibatch sampling noise. A decaying step can let the curve continue below its constant-step floor. At larger η, the approximate scaling can fail. Interpolating models or vanishing gradient noise can behave differently even at constant step size.