L1.4 More Data, a Different Model, or More Training? Learning Curves

Three proposals and one budget

Before launch, performance is insufficient and three colleagues propose changes:

A: label ten thousand more examples, taking months and substantial cost.
B: build a model ten times larger, increasing implementation and training costs.
C: train longer, which is useful only if the extra computation improves the goal.

Can smaller controlled experiments reveal which investment is promising? Learning curves provide evidence, not guaranteed predictions.

Train on subsets to inspect error versus sample count. Likewise vary capacity and update count, controlling the other choices and recording both training and validation scores.

This closes the generalization unit by turning fitting, bias–variance, splits, and CV into practical comparisons.

Which resource is missing: data, capacity, or training time?
Can a smaller experiment inform the expense?
How uncertain is that evidence?

DiscussionQ1

What are the three horizontal axes, and which curve features help assess more data?

Show the reasoning

Plot training and independent validation loss against each axis:

Curve	Horizontal axis	Proposal
Sample-size curve	Training count $n$	More data
Capacity curve	A specified flexibility setting	Different model
Training curve	Update count	More training

For sample-size curves, inspect

The gap and slope: a large gap with improving validation can motivate testing more data or regularization; it does not uniquely identify variance.
The apparent plateau: scores meeting far above a relevant baseline can motivate checking features, capacity, and optimization. A short flat segment does not prove more data can never help.

Use subsets such as $n/8,n/4,n/2,n$ , with repetitions and uncertainty, before making a costly extrapolation.

Inspect where the curves appear to meet, then test the assumptions behind that interpretation.

Shapes suggest experiments

Two common sample-size patterns are

High similar plateaus: investigate restricted capacity, weak features, objective mismatch, or insufficient optimization.
A substantial gap with improving validation: compare additional relevant data, regularization, or a more suitable family.

Capacity curves may show the classical tradeoff in Memorizing Old Exams: Overfitting and Underfitting, but need not be U-shaped universally. Their best configuration can change with sample count.

For training curves, both scores decreasing can motivate more updates. Falling training loss with worsening validation can motivate early stopping. A flat training curve warrants optimization and capacity checks, but can also reflect a sensible optimum or noise floor.

Irreducible population error and strong baselines help interpret plateaus. Repeated measurements and expert agreement can inform that reference with appropriate caveats. Training loss itself has no universal hard noise floor.

Sample-size, capacity, and training-duration curves with reference baselines.

The three axes answer different questions; reference levels and uncertainty make their trends interpretable.

Connect the shapes to risk

For squared loss, the repeated-training decomposition gives

\begin{aligned} \mathbb E[R(\hat f_n)] &=\underbrace{\mathbb E_X\operatorname{Var}(Y\mid X)}_{\text{irreducible}}\\ &\quad+\underbrace{\mathbb E_X\operatorname{bias}_n(X)^2}_{\text{mean prediction error}}\\ &\quad+\underbrace{\mathbb E_X\operatorname{Var}_D(\hat f_n(X))}_{\text{training variability}}. \end{aligned}

Both bias and variance may depend on $n$ . A stable positive bias plus decreasing variance can produce a plateau, but the identity does not force that pattern. Training–validation gap is not identical to prediction variance or estimation error.

\text{validation loss}\ \searrow\ \sigma^2+\operatorname{bias}^2\ \nwarrow\ \text{training loss} \qquad\text{(an illustrative pattern, not a general identity)}

Subset curves can suggest diminishing returns or modeling limitations, but do not prove an eventual plateau from a few observations. Power-law fits can summarize empirical scaling regimes [2]; extrapolation outside them is an additional hypothesis.

CautionTwo extrapolation traps

Fitting a power law outside its regime: small-data behavior, changing optimization, or a nonzero floor can bend a log–log curve. Several measured points and held-out checks are stronger evidence than extending a line through two points; there is no magic count that guarantees a valid extrapolation.

New data are different: subset curves describe more data drawn like the existing sample. New times, sources, or annotators can change the distribution and labeling process.

Compare performance on a recent representative batch as well. A mismatch motivates checking distribution, labeling, processing, and sampling uncertainty rather than attributing the result solely to quantity.

Prioritize the proposals

Check inexpensive explanations first: inspect optimization progress and useful baselines. More updates are a controlled experiment, not automatically the cheapest option at every scale.

Run a small-subset fit: reduce constraints temporarily and inspect gradients, loss, and capacity. Passing validates that limited test, not the entire task or generalization pipeline.

Then vary sample count: repeated subset experiments can motivate comparing added data against modeling changes. Estimate gains with uncertainty rather than treating the observed slope as an exact forecast.

State which procedure the curve measures. Holding hyperparameters fixed estimates that fixed configuration; retuning within each subset estimates an adaptively selected procedure and requires nested evaluation. Neither curve is inherently wrong, but they answer different questions. Keep evaluation independent and subsets representative.

Measured curves

Sample count: compare degree two and degree twelve on the shared toy. Representative medians over sixty repetitions are

n     d=2 train  d=2 risk   d=12 train  d=12 risk
10    0.1393     0.2904     —           —
20    0.1720     0.2493     0.0071      3.7113
40    0.2007     0.2409     0.0138      0.0458
80    0.2125     0.2275     0.0181      0.0282
160   0.2129     0.2246     0.0205      0.0247
320   0.2162     0.2227     0.0214      0.0235
640   0.2182     0.2213     0.0219      0.0229

The two procedures show very different returns from additional examples.

Degree two approaches a risk around 0.22, far above noise 0.0225. Increasing from 160 to 640 brings little improvement. Its restricted family motivates a modeling change rather than assuming another large annotation investment is best.

Degree twelve improves strongly, approaching risk 0.0229 at 640 examples. In this controlled toy, little population MSE remains above the known floor. At twenty examples, its thirteen coefficients are fewer than observations, but estimation is still unstable.

At twenty examples, degree two wins this comparison; at forty, degree twelve wins. Data quantity and useful capacity should therefore be planned together.

Training duration: use thirty training and 120 validation examples with a degree-twenty-nine model under fixed-rate full-batch gradient descent:

updates     training   validation   population risk
100         0.0713     0.1167       0.1317
1000        0.0516     0.0905       0.1015
10000       0.0302     0.0476       0.0502
100000      0.0196     0.0290       0.0290
1000000     0.0147     0.0324       0.0318
5000000     0.0130     0.0434       0.0458

Selected validation minimum: update 179270, validation 0.0284, population 0.0277.

Training loss keeps decreasing while fresh-data risk first improves and later worsens. Optimizing the training objective longer is not equivalent to optimizing population performance.

The architecture, observations, loss, and rate stay fixed. Duration alone changes the fitted function and its sensitivity to the training sample.

Training duration can act as a regularization choice.

Early stopping selects a checkpoint using validation behavior, often with patience or other rules to handle noise. It adds an evaluation and selection cost, even without changing the architecture.

That selection makes the winning validation score optimistic too. Use independent final evaluation. The toy’s known population mechanism exposes the difference directly.

Evaluation uncertainty: for a fixed model with risk 0.0258, repeated independent validation samples give representative results:

validation n=15:  mean 0.0257, SD 0.0092, approximate central width 0.0368
validation n=60:  mean 0.0260, SD 0.0046, approximate central width 0.0185
validation n=240: mean 0.0259, SD 0.0023, approximate central width 0.0094

The fixed-model sample means are unbiased under iid evaluation, but their uncertainty can exceed the trend being interpreted. These displayed repeated-sampling summaries are not automatically confidence intervals for every individual realized score.

A curve also invites selecting among many noisy points. Compare paired changes and appropriate uncertainty; overlapping error bars alone are not a general significance test. More evaluation data or a suitable resampling design may improve evidence before changing the model.

Compare the curves with uncertainty

Each axis asks a different resource-allocation question.

Check your understanding

References

Ng, A. Machine Learning Yearning 2018, Ch. 20–32.
Hestness, J. et al. Deep Learning Scaling is Predictable, Empirically. arXiv 2017.
Prechelt, L. Early Stopping — But When? Neural Networks: Tricks of the Trade, Springer 1998.