Learning goal: choosing the best validation score among m candidates can introduce systematic selection optimism. In an independent Gaussian-noise approximation, it grows roughly as √log m and scales with the validation noise σ. The exact behavior depends on candidate dependence and the search procedure.

Candidate scores on one axis; vertical: true-quality reference; highlight: selected candidate; arrow: optimism.

Selected validation score and true quality versus m, with a Gaussian extreme-value reference.

m = — Validation noise σ = — Selected validation score = — Selected model's true quality = — Optimism = — Gaussian reference σ√(2 log m) = —

Observe: with equal-quality candidates, increasing m tends to place the selected score farther from the true-quality reference. Increasing validation size fourfold can make the noise-related optimism roughly half as large (σ ∝ 1/√n): under the assumed sampling model. Independent outer evaluation estimates the selection procedure without reusing those observations for selection. It still has finite-sample uncertainty.