L2.2 When the Model Produces Its Own Targets

The target moves too

Ordinary labels stay fixed as parameters change. Without enough labels, could a model learn from its own prediction on a related input? Two students agreeing more often does not necessarily mean they become more correct.

Examples of model-defined targets include

• Self-distillation or mean teachers using model predictions as targets.
• Consistency between two views intended to share an answer.
• Bootstrap objectives containing current predictions in part of the target.

A simple consistency objective contains the model twice:

Every constant function makes it zero, a global minimum of this nonnegative objective. Agreement alone cannot distinguish informative representations from constant ones.

Three mechanisms, three roles

An anchor: nonconstant external labels or a fixed teacher constrain answers. Constant predictions cannot fit differing labels perfectly, but overall optima still depend on capacity, weighting, and the full objective.

EMA target: $\theta^-_{k+1}=\tau\theta^-_k+(1-\tau)\theta_k$ slows the reference’s response to single updates. This can reduce rapid chasing, but supplies neither correctness evidence nor universal stability.

Stop-gradient: blocks this update’s differentiation through the target branch. Its effectiveness interacts with predictors, normalization, view construction, and other constraints; alone it proves no universal noncollapse result.

The mechanisms act on different parts of the system. External labels are not a requirement of every successful self-supervised method.

Objective solutions and training destinations differ

Separate the following questions.

Whether constants minimize the forward objective is a property of the loss and allowed functions. Which solutions training approaches also depends on update dynamics. Stop-gradient can affect the latter without changing forward values.

Changing the minimizer set and changing the dynamics are different interventions.

For a linear update $\theta_{k+1}=(I-\eta A)\theta_k+\eta b$, eigenmode stability requires $|1-\eta\lambda|<1$. A nonsymmetric semigradient matrix can have complex or unstable eigenvalues; even positive symmetric curvature becomes unstable at excessive step sizes.

An EMA teacher adds another state, so analyze the joint online/target system. Slower targets can help in some regimes, but the joint eigenvalues, objective, and rates still determine stability.

Choose constraints and monitoring

Ask what limits the answers: true labels, a fixed teacher, view assumptions, predictors, normalization, or other information-preserving constraints. Decreasing consistency loss alone does not establish useful learning.

Choose $\tau$ against the intended target time scale, roughly $1/(1-\tau)$ updates. Values near one slow changes; schedules can adapt that speed during training. An appropriate value depends on the full procedure.

Monitor output variability, feature rank or singular values, and downstream performance. Reduced spread can warn of collapse without needing to shrink monotonically. Large spread alone does not establish usefulness.

Networks can collapse in only some dimensions. Confident pseudo-labels may help learning, but are not independent evidence proving their own correctness.

Compare mechanisms in one example

Use a degree-five polynomial, nearby pairs $(x,x+0.02)$, and three optional true-label anchors at $0.05,0.5,0.95$. The true sine function has range about two. For a clean comparison, use the full-gradient branch only with shared current parameters; EMA runs below use a stopped teacher:

import numpy as np
from ml_toy import truth
xs,ys = truth()
D = 5
mx = np.abs(np.vander(np.linspace(0,1,60),D+1,increasing=True)).max(0)
B = lambda X:np.vander(X,D+1,increasing=True)/mx
xa = np.array([.05,.5,.95])
ya = np.sin(2*np.pi*xa)
xp = np.linspace(0,1,200)
xq = np.clip(xp+.02,0,1)
Ap,Aq,Aa,As = B(xp),B(xq),B(xa),B(xs)
def train(stopgrad,anchor,tau,eta,T=20000,seed=0):
    if tau is not None and not stopgrad:
        raise ValueError('Use a stopped teacher for the EMA comparison.')
    w = np.random.default_rng(seed).standard_normal(D+1)*.5
    wt = w.copy()
    for k in range(T):
        ref = wt if tau is not None else w
        r = Ap@w-Aq@ref
        g = Ap.T@r/len(xp) if stopgrad else (Ap-Aq).T@r/len(xp)
        if anchor: g += 3.*Aa.T@(Aa@w-ya)/len(xa)
        w -= eta*g
        if tau is not None: wt = tau*wt+(1-tau)*w
        if not np.isfinite(w).all(): return None
    f = As@w
    return np.ptp(f),((f-ys)**2).mean()

Representative outcomes at eta=.02 after 20000 updates:
full gradient, no anchor:          range .077,       error .5374
stop-gradient, no anchor:          range 13017377,   error 1.13e14
stopped EMA(.99), no anchor:       range 34.187,     error 569.6434
full gradient plus anchors:       range .676,       error .3160
stop-gradient plus anchors:        range .671,       error .3136
stopped EMA plus anchors:          range .672,       error .3141

Interpret the rows separately.

Near collapse: full-gradient consistency produces a nearly flat range, far below the sine’s range. Small agreement loss is compatible with an uninformative output.

Divergence: the chosen one-sided semigradient and step produce enormous output, not collapse. Inspect the update matrix to identify its instability rather than assuming every stopped update diverges.

Slower instability: EMA reduces the displayed explosion but does not make the result correct. Smaller error than another failed run is not success.

Anchors: these rows are closer to truth yet remain too flat. Each objective term is averaged separately; sample-count ratio alone does not determine their relative weights.

The sine values at nearby positions are not exactly equal, so the consistency constraint itself is biased. The three anchors also miss peaks and troughs. Compare weighting, anchor coverage, and convergence before blaming only the target speed.

Average final weights: a separate use of EMA records a smoother evaluation model during otherwise unchanged SGD training:

wema = .99*wema+.01*w  # Initialize wema appropriately before the loop.

Representative population risks; direct fit .02744:
eta   B   final weights   EMA(.99) weights
.05   1   .02908          .02854
.05   8   .02831          .02739
.15   1   .03940          .02934
.15   8   .02893          .02752

In the displayed high-noise run, excess risk above the direct-fit reference falls by about 84%. That is one experimental comparison, not a universal reduction factor or proof of exact population optimality.

Polyak–Juditsky theory concerns cumulative averaging under specified asymptotic conditions, rather than every fixed-coefficient EMA [1]. Evaluate EMA separately for the intended model and task.

Compare anchors, EMA, and stopped gradients

Inspect representation spread and usefulness together with loss.

Check your understanding

References

1. Polyak, B. T., Juditsky, A. Acceleration of Stochastic Approximation by Averaging. SIAM Journal on Control and Optimization 1992.
2. Tarvainen, A., Valpola, H. Mean teachers are better role models. NeurIPS 2017.
3. Grill, J.-B. et al. Bootstrap Your Own Latent. NeurIPS 2020.
4. Chen, X., He, K. Exploring Simple Siamese Representation Learning. CVPR 2021.