Bias, variance, and irreducible error

Learning goal: expected squared error has three components. bias² measures how far the expected fitted predictor is from truth; variance measures variation across training samples; irreducible error is the conditional outcome variance, σ² in this toy. Model family, fitting, data size, and noise can influence the first two.

Thin: fits over R resamples; thick: their mean; dashed gray: truth.

Stacked by degree: σ², squared bias, and variance; dashed: their sum.

Degree d n

Noise σ Resamples R

bias² = — variance = — σ² = — Total = — Best tested degree d = —

Observe: increasing n can reduce the variance contribution while also changing finite-sample bias. At σ=0, variation in sampled input locations can still vary the fitted function. A small R makes the decomposition estimates noisy: estimating bias and variance also needs adequate replication.