Learning goal: if a hyperparameter makes optimized training error monotonically improve, selecting by training error favors the best-fitting end of that candidate list. It need not select the best generalization setting. Training duration can show a similar effect.

Training error, validation error, and known toy population risk versus polynomial degree; markers: selected settings.

The same comparison over training steps on a log axis.

Degree selected by training error = — Its population risk = — Degree selected by validation = — Its population risk = — Difference = —

Observe: varying the candidate limit often leaves the training-selected setting at the best-fitting boundary. This depends on nested families and fitting quality. A small validation set makes its selected setting noisy, and validation can also select a boundary. The lower view illustrates why further reduction of training error alone does not establish a useful stopping point.