L3.0 Recipe or Cooking Adjustment? Parameters and Hyperparameters

Written recipe and adjustments at the stove

A recipe specifies ingredients and amounts, rather than every tiny movement at every second.

Some choices define the cooking procedure; other adjustments happen while following it.

Likewise, weights are updated inside training while depth, rate, and stopping rules define or control that run. More advanced procedures can learn such settings too, so the distinction is relative to the chosen inner and outer loops.

Cooking smoothly and tasting good with fresh ingredients are different goals. Are we choosing the lowest training loss or the best future predictions? One measurement need not answer both.

What does the inner optimizer change?
What defines the procedure around it?
Which score answers the selection question?

DiscussionQ1

Classify the settings and explain why training loss alone cannot determine appropriate capacity.

Show the reasoning

The distinction is which procedure chooses the quantity, rather than how important it is.

\hat\theta(\lambda)=\arg\min_{\theta}\ \widehat R_n\big(f_\theta;\lambda\big).

Parameters $\theta$ are the inner fitted variables. Hyperparameters specify the objective, family, or solver around that fit. A learning rate need not appear in the ideal argmin to control its numerical approximation.

Two useful, overlapping roles are

	Family or regularization	Optimization procedure
Examples	Degree, depth, width, penalty, dropout	Rate, batch, momentum, gain, update count
Checks	Appropriate held-out comparisons and learning curves	Loss progress, gradients, scale, stability, and budget

For nested $\mathcal F_1\subseteq\mathcal F_2$ , the exact minimum training risk over the larger set is no greater: it can reuse the smaller set’s solution.

Thus maximal capacity is among training-score minimizers, possibly tied with others. Numerical and optimization errors can break observed monotonicity. What the score omits is the fresh-data cost of choosing extra flexibility.

Diagnose trainability, then compare generalization

First establish that candidate procedures can train meaningfully; then assess the question of interest.

Inspect stable progress, gradient scales, and small-subset fitting before attributing poor outcomes to capacity. An unstable run cannot provide a clean architecture comparison.

Use development evaluation to compare capacity and restrictions. Learning-curve patterns guide interventions but do not uniquely diagnose their cause.

Several settings affect both roles:

Duration is a compute budget and can act as implicit regularization.
Rate changes convergence and finite-time solutions.
Batch size affects update cost, count, noise, and possibly selected solutions.

Calling these “optimization settings” does not justify using training scores alone for a generalization claim.

Inner fitting and outer selection have different roles.

Training diagnostics and fresh-data comparisons answer different questions and interact.

Selecting capacity with training loss

Repeatedly split 120 toy examples into sixty training, thirty validation, and thirty test observations. Compare choosing degrees one through fifteen by training versus validation loss.

# Core selection step; x, y, tr, va, DEG, and Polynomial are set by the experiment.
fits = [Polynomial.fit(x[tr],y[tr],d) for d in DEG]
train_scores = [((f(x[tr])-y[tr])**2).mean() for f in fits]
val_scores = [((f(x[va])-y[va])**2).mean() for f in fits]

Representative 200-repeat record:
training selection: mode degree 15, upper limit selected 100%, median risk .0530
validation selection: mode degree 5, upper limit selected 3%, median risk .0262

The observed training selection favors the upper bound, as expected from nested-family minima. Extending past interpolation can produce ties or numerical problems rather than guaranteed unique selection of the highest degree.

Validation selection improves median population risk here. Its preferred degree is consistent with earlier toy comparisons, but finite validation noise and selection optimism remain.

Training fit alone contains no explicit cost for choosing additional capacity.

SupplementWhy can parameters be fitted on training data?

The distinction is not that every parameter fit is safe while every outer choice is forbidden on training data.

A restricted family, regularizer, and algorithm provide inductive bias for fitting weights. Choosing that restriction itself by unadjusted fit favors flexibility. Both levels need generalization reasoning; parameter fitting also overfits without suitable controls.

Complexity-corrected criteria can compare families under their assumptions; held-out evaluation supplies another approach. Raw training loss alone does not account for selection flexibility.

What about optimizer settings?

Could rates and update counts be selected using training loss because the formal function family stays fixed?

The answer depends on the question being answered.

For training-objective efficiency, comparing achieved training loss under a specified compute budget is appropriate. The known least-squares optimum offers a useful reference in the earlier optimizer example.

For fresh-data performance, settings influencing implicit regularization or the attained solution require suitable development evaluation.

Duration illustrates the difference directly:

updates   training   validation   population
100       .0713      .1167        .1317
10000     .0302      .0476        .0502
100000    .0196      .0290        .0290
1000000   .0147      .0324        .0318
5000000   .0130      .0434        .0458

In this record, longer training keeps improving fit while population risk worsens after an intermediate minimum. Training-score selection would miss the relevant generalization optimum.

Rate lacks a nested-family monotonic law: too large can destabilize and too small can leave incomplete fitting. Use training diagnostics to screen ineffective solves, then independent development evidence for generalization.

Optimizing a training score and selecting a useful predictor are different tasks.

State the target explicitly before choosing the metric: compute efficiency or fresh-data prediction quality?

Keep evaluation independent and rerun the permitted pipeline for each candidate. Nested-family monotonicity does not compare arbitrary unrelated architectures. Small validation sets retain uncertainty, regardless of how clearly the conceptual distinction is stated.

A practical sequence

Combine optimization and selection in a controlled loop:

Establish stable fitting and sensible gradient/activation scales for an initial configuration.
Compare capacity and regularization using development data or appropriate CV.
Recheck optimization after changing the family or representation.
Freeze the selected procedure before independent final evaluation.

Settings interact, so one-at-a-time tuning can miss useful combinations. The next note treats the search space itself.

Two roles, two evaluation questions

Compare training-based and validation-based capacity selection in the specified toy.

Check your understanding

References

Bergstra, J., Bengio, Y. Random Search for Hyper-Parameter Optimization. JMLR 2012.
Hastie, T., Tibshirani, R., Friedman, J. The Elements of Statistical Learning, 2nd ed. Springer 2009, Ch. 7.2, 7.10.
Dodge, J. et al. Show Your Work: Improved Reporting of Experimental Results. EMNLP 2019.