Learning goal: after selecting the best of m validation scores, that selected score need not remain an unbiased quality estimate. It is an extreme of noisy measurements. More candidates and fewer validation observations can increase optimism. With identical true candidate quality, independent test performance does not systematically improve merely from favorable validation noise.

Across 600 repetitions: selected validation score, selected model's test score, and average candidate quality; vertical lines: medians.

Median validation–test optimism versus candidate count, for three validation sizes.

Observe: with equal-quality candidates, increasing m makes selected validation scores more favorable without improving true candidate quality. Independent test estimates still fluctuate. Larger validation samples reduce noise and often reduce the selection gap: the effect scales with the validation noise in this experiment. The familiar √log m approximation requires suitable noise assumptions and is not universal for adaptive or correlated searches.