Learning goal: likelihood and plausible-looking samples measure different aspects of a generator. Adding a one-percent anomalous mixture component increases population NLL by at most log(1/0.99) ≈ 0.01 nat, while one percent of draws use that component. Conversely, concentrating on a plausible mode can produce attractive samples but very poor held-out likelihood. Evaluation should check both directions with metrics chosen for the application.

Dashed: target density; solid: contaminated and concentrated models; rugs: their generated samples.

Log horizontal axis: contamination ε. Left axis: excess NLL; right: sample-estimated W₂ distance.

ε = — Contaminated-model excess NLL = — Bound log(1/(1−ε)) = — Contaminated sample W₂ = — Concentrated-model excess NLL = — Concentrated sample W₂ = —

Observe: set ε to 0.01. The contaminated density remains close to the target near its main modes, and the displayed excess NLL is only 0.0066 nats. The 0.0101 bound is larger because the broad component still contributes some density near target observations. Meanwhile, sample W₂ rises from a 0.33 finite-sample reference to 1.03 in this run. About one percent of draws come from the broad component. A small likelihood penalty can coexist with an important anomaly rate. In the other direction, reduce the retained width to 0.1. Samples concentrate near a target mode; their W₂ here is 0.81, while excess NLL rises to about 43.9 nats because observations elsewhere receive tiny density. Complementary measurements help distinguish these failures; this does not prove every scalar score is useless.