Learning goal: the chain rule gives a product of derivatives. The loss is scalar, so reverse mode propagates vector–Jacobian products and obtains the full parameter gradient in one reverse sweep. Forward mode needs P coordinate directions for that full gradient. A separate issue is finite-difference accuracy: the best step is not necessarily h→0.

Cost of a full gradient versus parameter count, shown as reverse sweeps versus coordinate finite differences.

Central-difference relative error versus h, with truncation ∝h² and rounding amplification ∝1/h.

Parameter count P = — Reverse mode, one sweep = — Finite differences, 2P passes = — Ratio = — Relative error at current h = —

Observe: reverse mode uses one reverse sweep, while coordinate finite differences require a pass count that grows linearly with P. One sweep itself becomes more expensive as the network grows. The lower V-shaped curve combines truncation error proportional to h² with rounding amplification roughly proportional to 1/h. The minimum is near 10⁻⁵ in this numerical setup. Deliberately omitting the activation derivative raises the discrepancy to around 10⁻¹ here, showing how gradient checking can reveal a backward-pass bug.