Learning goal: Maximizing population log-likelihood, minimizing cross-entropy, and minimizing true-to-model KL are equivalent under the stated integrability conditions. Cross-entropy and KL differ by true entropy. In practice, the available quantity is empirical sample NLL, a fluctuating estimate of population cross-entropy whose variability depends on sample size.

Dashed blue: truth; rug: samples; solid red: current model; dashed green: empirical MLE.

Empirical NLL, population cross-entropy, and KL versus μ; horizontal reference: true entropy H(p).

Empirical NLL optimum μ̂ = — Population cross-entropy optimum = — Minimum cross-entropy H(p,q) = — True entropy H(p) = — Minimum KL = H(p,q) − H(p) = — |μ̂ − population optimum| = —

Observe: population cross-entropy and KL differ by a constant everywhere. Their minima occur at the same μ. Empirical NLL is the available sample estimate. Reducing n from 400 to 5 can make its minimizing μ fluctuate substantially. Larger samples typically reduce that uncertainty under appropriate assumptions. With bimodal truth and a restricted unimodal model, the minimum population KL is positive. More data cannot remove this family restriction.