M5.2 Tracking a Moving Dog: EMA, Moving Targets, and Stop-Gradient

Builds on2 notes

A dog that will not stand still

A tracker reports a running dog’s position each second, with independent errors of standard deviation about three meters. You want its current position.

Following each reading makes the estimate jump. Averaging ten seconds reduces noise but includes positions the dog left long ago, causing delay.

Can old observations lose influence gradually without storing the entire history?

DiscussionQ1

Define true position, noisy observation, and a recursive rule controlling memory.

Show the reasoning

Let $\mu_t$ be true position and $x_t=\mu_t+\varepsilon_t$ , with independent centered noise of variance $\sigma^2$ . We want an estimate close to the moving signal without inheriting all observation noise.

Instead of a stored window, keep one state:

m_t=\beta\,m_{t-1}+(1-\beta)\,x_t,\qquad\beta\in[0,1).

This exponential moving average (EMA) mixes the previous estimate and latest observation in proportions $\beta$ and $1-\beta$ . At $\beta=0$ it follows the latest reading; near one it changes slowly.

A leaking bucket

Each update retains fraction $\beta$ of the old state and adds $(1-\beta)x_t$ . An observation’s weight decays geometrically with age, with no hard cutoff.

At $\beta=0.9$ , a contribution retains about 35% of its original weight after ten updates; at $0.99$ , memory is much longer. The rough time scale is $1/(1-\beta)$ updates. Unlike a sliding window, storage remains constant.

EMA as geometric weights and a leaking bucket.

A single recursive state encodes geometrically weighted history.

One coefficient, two costs

Expand the recursion:

m_t=\beta^t m_0+(1-\beta)\sum_{k=0}^{t-1}\beta^k x_{t-k}.

The first term retains initialization. Only after $\beta^t$ becomes small do the infinite-history formulas apply. The stationary weighted mean age is $\beta/(1-\beta)$ , approximately $1/(1-\beta)$ only near one.

By linearity, the stationary estimate separates into weighted signal $(1-\beta)\sum_{k\ge0}\beta^k\mu_{t-k}$ and weighted noise $(1-\beta)\sum_{k\ge0}\beta^k\varepsilon_{t-k}$ .

Noise: independence makes the variance the sum of squared weights times $\sigma^2$ :

\operatorname{Var}(\text{noise})=\sigma^2(1-\beta)^2\sum_{k\ge0}\beta^{2k} =\sigma^2\frac{1-\beta}{1+\beta}.

At $\beta=0.9$ , standard deviation is about $0.23\sigma$ , reducing three meters to roughly 0.7 m.

Lag: for constant velocity $v$ , $\mu_{t-k}=\mu_t-vk$ . The weighted signal is $\mu_t-v\beta/(1-\beta)$ . With $v=2$ m per update and $\beta=0.9$ , the estimate lags by 18 m.

These effects oppose each other:

\underbrace{\sigma^2\frac{1-\beta}{1+\beta}}_{\text{noise variance decreases with }\beta} \quad\text{and}\quad \underbrace{\left(v\frac{\beta}{1-\beta}\right)^2}_{\text{squared lag increases with }\beta}.

For this model, faster signals favor smaller $\beta$ and noisier observations favor larger $\beta$ . Minimize their sum for the intended use; no coefficient is universally optimal. The ratio $v/\sigma$ determines the stationary tradeoff.

DetailsInitialization, Adam correction, and averaging iterates

Starting at m₀=0 gives observation weights summing to 1−β^t. For a constant-mean signal, dividing by this quantity removes the initialization shrinkage: Adam’s bias correction [3]. This does not generally remove tracking bias for a changing mean.

Parameters can also be averaged to reduce fluctuations. The classical Polyak–Juditsky analysis concerns arithmetic averaging of iterates, not every fixed-coefficient EMA setup [2].

A different issue: chasing your own output

The dog’s true position does not respond to the estimate. Now consider targets defined by the estimator itself.

A bootstrap target uses predictions to define training targets. Examples include Q-learning targets involving a next-state value estimate and self-supervised consistency between different views, such as BYOL [4].

If the only demand is equal outputs, a constant function can attain zero loss without distinguishing inputs. Not every bootstrap objective has this degeneracy: rewards or other constraints can restrict solutions.

Two mechanisms address different aspects of this feedback:

Stop-gradient: $\|f_\theta(a)-\mathrm{sg}[f(b)]\|^2$ treats the target branch as constant during differentiation, preventing that branch from moving through this gradient.
Slow target: use EMA parameters $\bar\theta_t=\beta\bar\theta_{t-1}+(1-\beta)\theta_t$ . Deliberate lag creates a slower reference. Periodically copied target networks, as in DQN [5], provide another update schedule.

Stop-gradient determines this update’s differentiation path; EMA determines target-parameter dynamics. Neither alone universally excludes collapse. Full systems such as BYOL include additional structure whose combined behavior needs analysis.

Put numbers on the tradeoff

For $v=2$ and $\sigma=3$ , scanning the stationary MSE gives an optimum near $\beta=0.35$ : lag about 1.1 m and noise standard deviation about 2.1 m. A stationary signal favors increasingly long averaging in this ideal independent-noise model, with finite-time initialization still relevant.

The formulas assume approximately constant signal velocity, independent centered noise, and an external target. Turns invalidate the constant-velocity lag expression; systematic measurement bias persists. Self-defined targets introduce the separate feedback issue above.

Simulate a turn and a trivial target

import numpy as np
rng = np.random.default_rng(0)
T, v, sigma = 200, 2., 3.
mu = v*np.arange(T)
x = mu+sigma*rng.standard_normal(T)
def ema(x,beta):
    m = np.empty_like(x)
    m[0] = x[0]
    for t in range(1,len(x)):
        m[t] = beta*m[t-1]+(1-beta)*x[t]
    return m
for beta in [0.,.5,.9]:
    err = ema(x,beta)[50:]-mu[50:]
    print(beta,err.mean(),err.std(),-v*beta/(1-beta),sigma*np.sqrt((1-beta)/(1+beta)))
mu2 = np.where(np.arange(T)<100,mu,2*mu[100]-mu)
m = ema(mu2+sigma*rng.standard_normal(T),.9)
print('turn error:',np.abs(m-mu2)[100:130].max())
theta = 5.
for _ in range(100):
    target = theta
    theta -= .1*2*(theta-target)
print('identical prediction and target:',theta)

The constant-velocity estimates agree approximately with theoretical lag and noise. After reversal, a transient develops before the lag approaches its new sign; the steady formula is not an instantaneous description at the turn. The final scalar example has zero loss and zero update at every value, illustrating that self-agreement alone teaches nothing, regardless of stopping the target gradient.

Vary $\beta$ , signal speed, and noise to inspect smoothing versus delay.

Check your understanding

References

Brown, R. G. Smoothing, Forecasting and Prediction of Discrete Time Series. Prentice-Hall, 1963.
Polyak, B. T., Juditsky, A. B. Acceleration of Stochastic Approximation by Averaging. SIAM Journal on Control and Optimization 30(4), 1992.
Kingma, D. P., Ba, J. Adam: A Method for Stochastic Optimization. ICLR 2015.
Grill, J.-B. et al. Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning. NeurIPS 2020.
Mnih, V. et al. Human-level Control through Deep Reinforcement Learning. Nature 518, 2015.