L3.3 Three Runs, Three Answers: What Should We Report?

Builds on1 note

M1.0Two Scales: Linear Combinations of Gaussians

Three different results

Three seeds give accuracies 0.831, 0.847, and 0.822.

Which result should be reported?

The best value describes a selected run; the average estimates a mean; the complete results reveal variation. These answer different questions.

A baseline’s three runs give 0.818, 0.835, and 0.809. Comparing only maxima suggests a 1.2-percentage-point advantage. What does it establish?

Within-method spread is substantial, but overlapping ranges alone neither prove nor disprove a method difference. Analyze the intended comparison and its uncertainty.

We will simulate apparent advantages between identical methods and derive what information is needed to plan replications.

Do we want typical performance or a selected best run?
How variable is the method difference?
Which repeated experiment represents the deployment question?

DiscussionQ1

What does each summary estimate, and why does naming the observed better result a winner create an apparent advantage?

Show the reasoning

A run’s score $X$ varies with specified randomness: initialization, batch order, augmentation, sampling, and possibly nondeterministic arithmetic. Choose whether the target is its mean, median, or a selection procedure’s performance.

Common summaries differ:

An iid finite-mean sample average is unbiased; with finite variance its standard error is $\sigma/\sqrt k$ . A sample median is not generally unbiased, and its usual root- $k$ asymptotics need regularity near a unique median.
A best-of- $k$ statistic targets an extreme, not a center. More trials shift it toward the favorable tail.

Two identical methods’ independently selected best scores still have zero expected signed difference. Calling the observed better one the winner changes the reported comparison to a nonnegative magnitude. This does not imply extrema always increase variance; the example’s apparent gap decreases with $k$ .

An observed winner alone is therefore uninformative about a reliable effect. Estimate the distribution of the method difference and define the effect of practical interest.

Measure run variability

Any uncertainty calculation needs the relevant variation first.

The earlier small tanh-network experiment records two hundred finite-training losses:

median .0034, standard deviation .0010, range [.0023,.0085]

The standard deviation is about 29% of the median in this particular MSE experiment. It characterizes this outcome scale, not every metric’s precision.

Variability differs greatly across tasks and randomness sources. A few repetitions give preliminary evidence, but tail-sensitive spread estimates may need more. Measure the intended project rather than transferring a generic percentage.

Two identical outcome distributions can produce different selected best runs.

Repeated-run summaries and comparison design determine what an apparent advantage means.

Apparent advantage from selecting observed winners

Draw both methods from the same empirical two-hundred-run distribution. Each chooses its minimum loss among $k$ draws. Repeating the comparison gives the following relative winner-versus-loser gaps:

k    median apparent gap   90th percentile
1    16.9%                 70.1%
3    10.5%                 26.8%
5    8.9%                  23.3%
10   6.9%                  19.1%

At three trials, 10.5% is the median apparent gap under this particular MSE distribution, not a mean or a threshold transferable to accuracy.

The best observed run does not estimate a method’s typical improvement.

Here gaps shrink with $k$ . For bounded-below losses, minima can converge to a support endpoint. The issue is the object being estimated, not that extreme statistics can never converge.

A mean or median instead targets a center, with uncertainty shrinking under the relevant assumptions. Choose the target before increasing trial count.

CautionA seed does not capture every source of randomness

Training randomness is only one layer:

Dataset generation or splitting can change results.
Augmentation, dropout, and batch sampling introduce further randomness.
Nondeterministic parallel arithmetic can vary even with the same seeded random draws.

Match repetitions to the target. Conditional training performance on one fixed dataset can legitimately hold it fixed. Estimating a resampled-data procedure needs data variability included. A deterministic mode can reduce some hardware variability, with platform-dependent costs and limits.

Plan repetitions using the difference

Let paired scores give $D_i=X_i^A-X_i^B$ . For independent replications with finite difference variance,

\bar D=\frac1k\sum_{i=1}^kD_i, \qquad\operatorname{SE}(\bar D)=\frac{\sigma_D}{\sqrt{k}}.

Halving the standard error requires roughly four times as many pairs. Confidence-interval width also depends on the chosen confidence procedure; testing power depends on effect size, significance level, and test design. A 29% single-method relative spread does not determine a universal seed count.

Specify the difference the experiment must resolve before spending the compute budget.

Pairing can help because

\begin{aligned} \operatorname{Var}(X^A-X^B) &=\operatorname{Var}(X^A)+\operatorname{Var}(X^B)\\ &\quad-2\operatorname{Cov}(X^A,X^B). \end{aligned}

Shared splits or matched randomness reduce difference variance when covariance is positive. Matching seed numbers alone does not ensure different architectures consume the same underlying random quantities.

What to report

Make the following distinctions visible:

The estimand and summary: mean or median, repetitions, and a suitable spread or uncertainty measure.
Paired differences where the design supports meaningful pairing.
Hyperparameter search effort and selection procedure, separately from final-run replications.
Any best-run statistic explicitly identified as selected, alongside typical performance.
A final evaluation protocol kept outside subsequent modeling decisions.

State what was repeated: training on fixed observations or the entire data-sampling pipeline. Range and interquartile range describe run spread; confidence intervals concern uncertainty in a specified estimand. Relative MSE changes and accuracy percentage points are not interchangeable.

Estimate variability for your experiment

First collect run outcomes, then method pairs if comparing procedures.

A short exploratory driver is

# run_everything defines and records the intended sources of randomness.
res = [run_everything(seed=s) for s in range(8)]
med = np.median(res)
spread = np.std(res,ddof=1)
print('median:',med,'sample SD:',spread)
# Report the individual scores; relative spread is meaningful only for a suitable scale.

Eight runs can reveal issues but do not guarantee precise tail or standard-deviation estimates. Retain paired outcomes, assess distribution shape, and plan more repetitions according to the desired uncertainty.

Compare summaries as repetitions increase

The demo’s ranking frequency is not a significance-controlled test power, nor a universal replication requirement.

Check your understanding

References

Dodge, J. et al. Show Your Work: Improved Reporting of Experimental Results. EMNLP 2019.
Bouthillier, X. et al. Accounting for Variance in Machine Learning Benchmarks. MLSys 2021.
Henderson, P. et al. Deep Reinforcement Learning that Matters. AAAI 2018.