Learning goal: changing the loss changes what the objective rewards. The same outlier affects squared, absolute, and Huber fits differently. A separate classification example shows how the loss changes gradients for confidently wrong predictions.

Points: data; red-circled outlier has adjustable height; dashed gray: truth; solid: squared, absolute, and Huber fits.

Horizontal: correct-class logit z. Vertical: magnitude of loss derivative with respect to z.

Observe: raise the outlier; the squared-error fit changes substantially here, while absolute and Huber fits are less affected. Below, a confidently wrong sigmoid prediction gives a vanishing squared-loss logit gradient but a substantial cross-entropy gradient. This changes optimization behavior; neither loss alone guarantees successful training.