U7.0 ~55 min Oct 2026

The model is trained. Now the user wants a dog in a red hat.

Predict, plot, and interpret evidence within each complete method route, then derive the details. Review the table last and reason about combined methods and new requirements.

Prefer continuous reading? Skill manual Read the note · U7.0 →

A new request arrives

Nori A polar bear in an orca hat

Suppose we spent two weeks training a diffusion model to generate dog photos. It can draw dogs with different coats, poses, and backgrounds, sometimes even wearing hats. After looking at the results, the user says: “This time, I want a dog in a red hat.”

On the left, varied dog photos from an already trained model. On the right, a request for a dog in a red hat, received only at use time. The right panel illustrates the request, not a generated answer. — The model might draw a red-hat dog, but it has not received this request. Our model has no text or class input, so we cannot simply supply “wearing a red hat” as a prompt.
Select the image to enlarge

You spread a few results on the workbench and invite three learning partners to help: Lumi, a winged deer; Mugi, a corgi with a Shiba face; and Nori, a polar bear in an orca hat. Each has a proposal.

What would you try first to make red-hat dogs appear more often? Prepare the data, then choose a method to explore fully. Each route reaches a training or generation implementation and a distribution plot before returning to compare another method.

Look at the data before choosing a route

Nori A polar bear in an orca hat

Replace two distinguishable appearances with one-dimensional data $x_0$ : the left class is centered at $-2$ , the right at $2$ , both with standard deviation $\sigma=0.55$ . The right class represents the current preference. Variation within a class remains meaningful; larger $x_0$ is not defined to mean better.

The curve is the known data-generating density. The dots are sampled training observations (32 of 256 per class shown). The right class represents the preferred appearance; x₀ is a teaching coordinate, not a measured dog-image feature.

The original data-generating distribution gives each class half the mass:

p_{\mathrm{base}}(x_0)=\tfrac12\mathcal N(x_0;-2,\sigma^2)+\tfrac12\mathcal N(x_0;2,\sigma^2).

$\mathcal N(x_0;\mu,\sigma^2)$ is a Gaussian density centered at $\mu$ , with spread set by $\sigma$ . The gray dashed curve always shows this reference base; the solid curve shows the current model or returned density.

Track the data, input $c$ , parameters $\theta$ , and return rule: which does this operation change? First operate a directly sampleable toy generator, then connect the evidence to the full diffusion / Flow Matching training objectives. Complete each route’s derivations and observation checks before marking it explored.

Choose a method and explore it fully

Choose your starting method. Each route includes the definitions, training or generation procedure, and a distribution-plotting implementation. Only reaching its ending marks it fully explored. You can return early and continue later.

Conditional training: learn to use descriptions

Teach the model to read descriptions

Lumi A winged deer

“We could collect photos with descriptions. For a dog in a red hat, give the model both the photo and those words. If it practices reading descriptions during training, we can tell it what we want during generation.”

Lumi wants to add an input: besides the image, the model should read a general condition, such as a class or text prompt, denoted by $c$ . Write a clean image as $x_0$ . Training data then consists of pairs $(x_0,c)$ , and the model learns how images are distributed given a description:

p_\theta(x_0\mid c).

Here $\theta$ denotes the generative model’s parameters. For continuous images, $p_\theta$ is a probability density, not the probability of drawing one exact image. Given “a dog in a red hat,” images matching that description should receive higher density.

Providing the condition during training so the model learns to generate accordingly is called conditional training. This does not mean retraining for every new prompt: a model that has learned text inputs may combine descriptions it has never seen together. Our original model, however, lacks that input and needs to acquire this ability first.

Try the operation on a plot first. There are $256$ labelled samples per class. This toy model uses input $c$ to choose a Gaussian mean $\mu_c$ ; its learnable parameters $(\mu_0,\mu_1)$ both start at zero and the spread stays fixed. Learn the means, then change only $c$ and inspect the plot and parameters.

Conditional training: two inputs, one parameter set

Lumi A winged deer

Predict before running A/B. Inspect which density changes location and whether μ₀ and μ₁ update. Then use the controls to try one training step and inspect incomplete learning.

Data: Left/right samples with class labels
Input c: Choose 0 or 1 at generation
Model parameters θ: This toy learns μ₀ and μ₁; switching c does not update them
Return rule: Sample the current model directly

Predict, then run a controlled comparison

After learning both means, what changes in the density and parameters when only c switches from 1 to 0?

Original mixture Training-data target Current model / return density Generated histogram

μ₀ = 0 · μ₁ = 0 · σ = 0.55

Compare the curves, then train or generate and inspect where the histogram falls.

Explanation of the standard A/B comparison

A and B use identical learned means and both record 60 updates. A uses c = 1 and B uses c = 0, moving the density from right to left. The unchanged update record supports no retraining. After only one step, both means remain near zero; switching c does not imply learning is complete.

Example: train the two means on 256 labelled samples per class. μ₀ approaches −2 and μ₁ approaches 2. Changing c then selects a mode without updating either mean.

After custom control or Python changes, use your actual plot and parameters; this standard comparison is not a record of that custom run.

This implementation samples Gaussian-mixture data, updates learnable parameters or selects candidates, and plots densities with 1,000 generated samples. This directly sampleable toy generator is not a trained diffusion network.

Edit Python and run the experiment yourself

Change the parameters or the sampling rule. Python loads when you press Run and executes in your browser. Each run starts fresh; your edits are not saved.

The first opening imports the controls above; reopening preserves your edits. Settings are imported, not accumulated trained parameters or samples, so finite-sample plots need not match exactly.

First change only c near the top and run. The four sections are data, training, generation, and plotting. Generation uses the just-trained parameters; rerunning the whole script starts training from the initial model again.

Experiment code (Python)

import math
import numpy as np
import matplotlib.pyplot as plt

# 1. DATA AND SETTINGS -- edit one setting above the model code.
c = 1  # EDIT: 0 = left, 1 = right
steps = 60
rng = np.random.default_rng(42)
sigma = 0.55
grid = np.linspace(-5, 5, 500)
def pdf(x, mu):
    return np.exp(-0.5 * ((x - mu) / sigma) ** 2) / (sigma * np.sqrt(2 * np.pi))
def sample_mixture(n, right=0.5):
    mu = np.where(rng.random(n) < right, 2.0, -2.0)
    return rng.normal(mu, sigma)
base = 0.5 * pdf(grid, -2) + 0.5 * pdf(grid, 2)

data = [rng.normal(mu, sigma, 256) for mu in [-2.0, 2.0]]

# 2. TRAINING -- each full Run starts from these initial means.
means = np.zeros(2)
for _ in range(steps):
    gradient = 0.5 * np.array([means[label] - data[label].mean() for label in [0, 1]])
    means -= 0.15 * gradient

# 3. GENERATION -- both inputs below use the SAME trained means.
for label in [0, 1]:
    print("same trained parameters, c =", label, "uses mean", means[label])
samples = rng.normal(means[c], sigma, 1000)
target = pdf(grid, [-2.0, 2.0][c])
after = pdf(grid, means[c])
print("Trained means:", means)

# 4. PLOTTING -- density curves and the actual returned samples.
plt.figure(figsize=(7, 3.8))
plt.plot(grid, base, color="gray", linestyle="--", label="original base")
plt.plot(grid, target, color="teal", linestyle=":", label="training-data target")
plt.plot(grid, after, color="darkorange", label="model / returned density")
plt.hist(samples, bins=np.linspace(-5, 5, 41), density=True, alpha=0.25, color="darkorange", label="1000 generated outputs")
plt.xlabel("x0")
plt.ylabel("density")
plt.xlim(-5, 5)
plt.legend(fontsize=8)
plt.tight_layout()
plt.show()

import math
import numpy as np
import matplotlib.pyplot as plt

# 1. DATA AND SETTINGS -- edit one setting above the model code.
c = 1  # EDIT: 0 = left, 1 = right
steps = 60
rng = np.random.default_rng(42)
sigma = 0.55
grid = np.linspace(-5, 5, 500)
def pdf(x, mu):
    return np.exp(-0.5 * ((x - mu) / sigma) ** 2) / (sigma * np.sqrt(2 * np.pi))
def sample_mixture(n, right=0.5):
    mu = np.where(rng.random(n) < right, 2.0, -2.0)
    return rng.normal(mu, sigma)
base = 0.5 * pdf(grid, -2) + 0.5 * pdf(grid, 2)

data = [rng.normal(mu, sigma, 256) for mu in [-2.0, 2.0]]

# 2. TRAINING -- each full Run starts from these initial means.
means = np.zeros(2)
for _ in range(steps):
    gradient = 0.5 * np.array([means[label] - data[label].mean() for label in [0, 1]])
    means -= 0.15 * gradient

# 3. GENERATION -- both inputs below use the SAME trained means.
for label in [0, 1]:
    print("same trained parameters, c =", label, "uses mean", means[label])
samples = rng.normal(means[c], sigma, 1000)
target = pdf(grid, [-2.0, 2.0][c])
after = pdf(grid, means[c])
print("Trained means:", means)

# 4. PLOTTING -- density curves and the actual returned samples.
plt.figure(figsize=(7, 3.8))
plt.plot(grid, base, color="gray", linestyle="--", label="original base")
plt.plot(grid, target, color="teal", linestyle=":", label="training-data target")
plt.plot(grid, after, color="darkorange", label="model / returned density")
plt.hist(samples, bins=np.linspace(-5, 5, 41), density=True, alpha=0.25, color="darkorange", label="1000 generated outputs")
plt.xlabel("x0")
plt.ylabel("density")
plt.xlim(-5, 5)
plt.legend(fontsize=8)
plt.tight_layout()
plt.show()

Package messages and warnings

Which evidence supports generation without retraining?

Lumi A winged deer

A/B first learns both means, then changes only $c$ . The density centers differ. Which evidence supports one model answering two inputs?

Neither learned mean nor the update count changed; c selects a different learned mean.

Different samples can come from fixed parameters. Inspect parameters and update counts directly.

Changing an input can change a generated distribution without updating parameters. A curve alone does not identify what changed.

Train a model that actually reads a class

Lumi A winged deer

Label the left group $c=0$ and the right group $c=1$ , with $m=256$ training examples per class. The toy conditional generator is $p_\theta(x_0\mid c)=\mathcal N(x_0;\mu_c,\sigma^2)$ with learned parameters $\theta=(\mu_0,\mu_1)$ . Both means start at zero, so merely supplying $c$ does not yet separate its outputs.

With fixed variance, maximizing Gaussian likelihood is equivalent to minimizing squared error:

\mathcal L_{\mathrm{toy}}=\frac{1}{2m}\sum_{c=0}^{1}\sum_{i=1}^{m}\frac12(x_0^{(i,c)}-\mu_c)^2.

The derivative for $\mu_c$ is $\tfrac12(\mu_c-\bar x_c)$ , where $\bar x_c$ is that class’s sample mean. The code updates both means using this gradient; generation then selects one by $c$ . This is a directly sampleable Gaussian model. Next, return to diffusion and Flow Matching: what must change in noise or velocity regression when a network also reads $c$ ?

Does a new input require a new target?

Lumi A winged deer

Lumi wants to keep U1’s noise-prediction exercise. Draw an image-description pair $x_0,c$ , choose noise level $t$ , and add known noise $\epsilon$ to obtain $x_t$ . The network now also sees $c$ . Which quantity should remain the regression target?

Keeping noise-prediction regression means keeping the added noise as the label; the condition is an extra input.

The description is information for the network to read, not the answer to this noise-prediction exercise.

The exercise predicts noise. A clean-image predictor is another possible design, but that changes more than just the condition input.

Add the description, keep the target?

Lumi A winged deer

“It was practicing noise prediction. Could we keep that exercise and just let it read the description too?”

Yes. Draw $(x_0,c)$ , choose a noise level $t$ , and add known noise $\epsilon$ to form $x_t$ . The network now sees $x_t,t,c$ and outputs $\epsilon_\theta(x_t,t,c)$ . We still compare it with the noise $\epsilon$ actually added.

The prediction error to minimize is therefore

\begin{gathered} \mathcal L_{\mathrm{diff,cond}}(\theta)\\ =\mathbb E_{x_0,c,t,\epsilon}\Big[\\ \|\epsilon-\epsilon_\theta(x_t,t,c)\|^2\Big]. \end{gathered}

The expectation averages over training images and descriptions, sampled times, and added noise. Squared error measures how far the prediction is from the known noise. Updating $\theta$ with this loss lets the model learn different predictions for different descriptions.

We keep diffusion’s time convention: larger $t$ means more noise, and generation moves toward the clean end at $t=0$ . The new element is the input $c$ . The regression target becomes neither text nor a finished image.

Can velocity prediction read descriptions too?

Lumi A winged deer

For Flow Matching, the original target is a reference path’s velocity. Write the noise endpoint as $X_0$ and data endpoint as $X_1$ , with condition $c$ attached to the data endpoint. Here time runs from the source at $t=0$ to data at $t=1$ , separately from diffusion’s convention above.

For example, choose a straight reference path:

\begin{gathered} I_t=(1-t)X_0+tX_1,\\ \partial_t I_t=X_1-X_0. \end{gathered}

Training endpoints are known, so this velocity label can be computed. Give $I_t,t,c$ to the velocity network and compare its prediction $v_\theta(I_t,t,c)$ with the same original target:

\begin{gathered} \mathcal L_{\mathrm{FM,cond}}(\theta)\\ =\mathbb E_{t,(X_0,X_1,c)}\Big[\\ \|v_\theta(I_t,t,c)-\partial_t I_t\|^2\Big]. \end{gathered}

This expectation averages over sampled endpoints, conditions, and times along the path. During generation, the data endpoint is unknown; we follow the learned velocity step by step. The reference path constructs training examples rather than assigning each generated sample a known endpoint or route in advance.

Lumi makes the same modification for both models: keep the original regression target and let the network also read $c$ . The condition affects $\theta$ during training. That is conditional training.

What stays when both models read descriptions?

Lumi A winged deer

Diffusion practices noise prediction; Flow Matching practices reference-velocity prediction. Does Lumi make the same kind of modification to both?

The networks predict different quantities, but both can read a condition while retaining their original targets.

Flow Matching’s target is reference velocity, not diffusion’s added noise. Adding a condition does not make those targets identical.

Known training endpoints allow construction of reference velocity. During generation, the endpoint and entire trajectory remain unknown.

Route complete: learned capability and generation-time input

Lumi A winged deer

The two means have learned from labelled data. With $\theta$ fixed, switching $c$ selects another conditional distribution. Return to the plot to try changing only the input.

Conditional training needs data associating observations with conditions, updates parameters during learning, and supplies that input at generation. A model that already understands text need not retrain for every comprehensible prompt; our original model first needs to acquire that input capability.

Learning means was a toy-model choice. Conditional training establishes the ability to use $c$ ; it does not restrict learnable parameters to means.

Fine-tuning: update the original model on new data

Continue learning from photos we like

Mugi A corgi with a Shiba face

“We already have a model that draws dogs. Let it draw a batch, then pick the red-hat photos we like. If it keeps practicing on those photos, might it draw what we prefer more often?”

Mugi wants to change the training data: start from the existing model and continue training on newly collected preferred images. This is fine-tuning. Since the original model does not read descriptions, this approach need not add a text input first.

The request arrives after the original training, but training still writes the preference into $\theta$ . Next time, we use the updated model. In a moment, we will write out which parts of the original exercise stay and how the parameters change.

First compare two operations on the original equal-mixture toy model. This version fixes centers and spread and lets parameter $\theta$ determine the right-mode weight; its parameterization and loss come afterward. Learn once from preferred data, then change only the data without updating. Does the model automatically forget its learned preference?

Fine-tuning: does changing data update the model?

Mugi A corgi with a Shiba face

Predict what happens when data and model updates are separated, then run A/B. The displayed loss evaluates the model on the current data; inspect it alongside the parameters before deriving its calculation.

Data: New preferred samples
Input c: This model has no c input
Model parameters θ: This toy learns only the mixing logit
Return rule: Sample the current model directly

Predict, then run a controlled comparison

Change preferred data from 90% right to 10% right without updating the model. Which change: the model weight, generated density, or loss on the current data?

Original mixture Training-data target Current model / return density Generated histogram

Original model: equal mixture · μ = −2, 2 · σ = 0.55

Compare the curves, then train or generate and inspect where the histogram falls.

Explanation of the standard A/B comparison

After A trains, B changes only the evaluation data to favor the left. θ, the model right weight, and update count stay fixed, so the model densities overlap. Loss also depends on evaluation data and can rise when that data changes. A changed loss alone does not establish changed θ.

Example: the original right weight is 0.5. Likelihood updates on roughly 90% right-mode preferred data move generated mass toward the right. Later generations retain the updated weight.

After custom control or Python changes, use your actual plot and parameters; this standard comparison is not a record of that custom run.

Edit Python and run the experiment yourself

Change the parameters or the sampling rule. Python loads when you press Run and executes in your browser. Each run starts fresh; your edits are not saved.

The first opening imports the controls above; reopening preserves your edits. Settings are imported, not accumulated trained parameters or samples, so finite-sample plots need not match exactly.

First change preferred_right near the top and rerun. Inspect whether training writes the preference into the weight. Each full run starts from logit = 0; it does not import the trained state from the controls above.

Experiment code (Python)

import math
import numpy as np
import matplotlib.pyplot as plt

# 1. DATA AND SETTINGS -- edit one setting above the model code.
preferred_right = 0.9  # EDIT: share in the new DATA
steps = 60
rng = np.random.default_rng(42)
sigma = 0.55
grid = np.linspace(-5, 5, 500)
def pdf(x, mu):
    return np.exp(-0.5 * ((x - mu) / sigma) ** 2) / (sigma * np.sqrt(2 * np.pi))
def sample_mixture(n, right=0.5):
    mu = np.where(rng.random(n) < right, 2.0, -2.0)
    return rng.normal(mu, sigma)
base = 0.5 * pdf(grid, -2) + 0.5 * pdf(grid, 2)

data = sample_mixture(512, preferred_right)

# 2. TRAINING -- update the original model on the preferred DATA.
logit = 0.0
for _ in range(steps):
    right = 1 / (1 + np.exp(-logit))
    density = (1 - right) * pdf(data, -2) + right * pdf(data, 2)
    responsibility = right * pdf(data, 2) / density
    gradient = right - responsibility.mean()
    logit -= 0.3 * gradient

# 3. GENERATION -- retain the learned weight; no new update here.
right = 1 / (1 + np.exp(-logit))
samples = sample_mixture(1000, right)
target = (1 - preferred_right) * pdf(grid, -2) + preferred_right * pdf(grid, 2)
after = (1 - right) * pdf(grid, -2) + right * pdf(grid, 2)
print("Updated logit:", logit, "right-mode probability:", right)

# 4. PLOTTING -- density curves and the actual returned samples.
plt.figure(figsize=(7, 3.8))
plt.plot(grid, base, color="gray", linestyle="--", label="original base")
plt.plot(grid, target, color="teal", linestyle=":", label="training-data target")
plt.plot(grid, after, color="darkorange", label="model / returned density")
plt.hist(samples, bins=np.linspace(-5, 5, 41), density=True, alpha=0.25, color="darkorange", label="1000 generated outputs")
plt.xlabel("x0")
plt.ylabel("density")
plt.xlim(-5, 5)
plt.legend(fontsize=8)
plt.tight_layout()
plt.show()

import math
import numpy as np
import matplotlib.pyplot as plt

# 1. DATA AND SETTINGS -- edit one setting above the model code.
preferred_right = 0.9  # EDIT: share in the new DATA
steps = 60
rng = np.random.default_rng(42)
sigma = 0.55
grid = np.linspace(-5, 5, 500)
def pdf(x, mu):
    return np.exp(-0.5 * ((x - mu) / sigma) ** 2) / (sigma * np.sqrt(2 * np.pi))
def sample_mixture(n, right=0.5):
    mu = np.where(rng.random(n) < right, 2.0, -2.0)
    return rng.normal(mu, sigma)
base = 0.5 * pdf(grid, -2) + 0.5 * pdf(grid, 2)

data = sample_mixture(512, preferred_right)

# 2. TRAINING -- update the original model on the preferred DATA.
logit = 0.0
for _ in range(steps):
    right = 1 / (1 + np.exp(-logit))
    density = (1 - right) * pdf(data, -2) + right * pdf(data, 2)
    responsibility = right * pdf(data, 2) / density
    gradient = right - responsibility.mean()
    logit -= 0.3 * gradient

# 3. GENERATION -- retain the learned weight; no new update here.
right = 1 / (1 + np.exp(-logit))
samples = sample_mixture(1000, right)
target = (1 - preferred_right) * pdf(grid, -2) + preferred_right * pdf(grid, 2)
after = (1 - right) * pdf(grid, -2) + right * pdf(grid, 2)
print("Updated logit:", logit, "right-mode probability:", right)

# 4. PLOTTING -- density curves and the actual returned samples.
plt.figure(figsize=(7, 3.8))
plt.plot(grid, base, color="gray", linestyle="--", label="original base")
plt.plot(grid, target, color="teal", linestyle=":", label="training-data target")
plt.plot(grid, after, color="darkorange", label="model / returned density")
plt.hist(samples, bins=np.linspace(-5, 5, 41), density=True, alpha=0.25, color="darkorange", label="1000 generated outputs")
plt.xlabel("x0")
plt.ylabel("density")
plt.xlim(-5, 5)
plt.legend(fontsize=8)
plt.tight_layout()
plt.show()

Package messages and warnings

Why can the loss change while model densities overlap?

Mugi A corgi with a Shiba face

The A/B data changes from roughly $90\%$ right to $10\%$ right. $\theta$ , model weight, and update count stay identical, but B has a higher loss. Which explanation agrees with that evidence?

Loss also depends on the evaluation data. A model retaining its right-class preference can have higher negative log-likelihood on left-heavy data.

The recorded parameters and update counts are identical. A changed loss alone does not establish another update.

Finite histograms fluctuate while the exact density and weight remain fixed. Sampling variation is not forgetting.

Actually update the original mixture's weight

Mugi A corgi with a Shiba face

Keep the component means and standard deviation fixed, and learn the original model’s mixture weight. Parameter $\theta$ specifies right-mode weight $w_\theta=1/(1+e^{-\theta})$ :

p_\theta(x_0)=(1-w_\theta)\mathcal N(x_0;-2,\sigma^2)+w_\theta\mathcal N(x_0;2,\sigma^2).

Originally $\theta=0$ , giving equal weights. Collect $512$ preferred examples, initially about $90\%$ from the right group. No $c$ is supplied. Update the same parameter using negative log-likelihood:

\mathcal L_{\mathrm{toy,FT}}(\theta)=-\frac1m\sum_{i=1}^m\log p_\theta(x_0^{(i)}).

The code computes $r_i=w_\theta\mathcal N(x_0^{(i)};2,\sigma^2)/p_\theta(x_0^{(i)})$ , the current model’s posterior responsibility for the right group. The gradient is $w_\theta-\frac1m\sum_i r_i$ . After updating $\theta$ , generate from the new $p_\theta$ and inspect whether mass moved toward the preferred data.

Fixing centers and learning only a weight is a choice made for this toy plot. Fine-tuning can also update centers, other network weights, or a full parameter set; the defining operation is continuing to learn from new data in the original model.

How do selected photos become practice?

Mugi A corgi with a Shiba face

Write Mugi’s selected preferred data as $\mathcal D_{\mathrm{pref}}$ . The original model does not read $c$ , so this dataset contains clean images $x_0^{(i)}$ without adding descriptions it cannot use.

Draw $x_0$ from these images, choose $t$ , and add $\epsilon$ to form $x_t$ . We still compare predicted noise with the noise actually added; only the image source changes:

\begin{gathered} \mathcal L_{\mathrm{align}} (\theta;\mathcal D_{\mathrm{pref}})\\ =\mathbb E_{\substack{x_0\sim\mathcal D_{\mathrm{pref}}\\t,\epsilon}}\Big[\\ \|\epsilon-\epsilon_\theta(x_t,t)\|^2\Big]. \end{gathered}

This loss measures noise-prediction error on preferred images, averaging over images drawn from that dataset, times, and noise. The preferred data changes, while the regression exercise stays the same.

For an already conditional model, collect $(x_0,c)$ and also supply $c$ to the prediction. Flow Matching can likewise retain velocity regression and practice on preferred data. In each case, the data distribution used to update the model changes.

A late request can still update parameters

Mugi A corgi with a Shiba face

With this loss, gradient descent can continue the model’s learning:

\begin{gathered} \theta\leftarrow\theta\\ -\eta\nabla_\theta \mathcal L_{\mathrm{align}} (\theta;\mathcal D_{\mathrm{pref}}). \end{gathered}

We differentiate with respect to $\theta$ because the learned parameters are what we want to change. The gradient points toward increasing loss, so the minus sign moves toward lower error; the learning rate $\eta$ sets the update size.

Mugi’s request arrives after the original training, but this update still writes it into $\theta$ . Tomorrow, even without the image-selection program, we would be using an updated model.

Other alignment methods may use pairwise comparisons or reward weighting. Here we chose a directly computable regression loss on preferred data. To identify the boundary, check whether the original $\theta$ changed, not just when the request arrived.

What is Mugi trying to change?

Mugi A corgi with a Shiba face

The update contains $\nabla_\theta\mathcal L_{\mathrm{align}}$ . For which object does this gradient provide an update direction?

The derivative’s subscript is θ, the learnable network parameters.

This derivative is with respect to θ, not the current image input. Those are different objects to modify.

Preferred photos construct training examples. The update changes network parameters rather than repainting the collected photos.

Must fine-tuning only change mixture weights?

Mugi A corgi with a Shiba face

This toy exposes only the mixing weight for learning to keep its plot easy to read. Now also make the Gaussian means learnable and continue updating the original model on preferred data. Is this still fine-tuning?

Fine-tuning means continuing to update the original model on new data, not restricting which parameter type can learn.

Conditional training establishes the use of input c. Having learnable means does not establish that input.

Steering preserves generative parameters. Updating means explicitly changes those parameters.

Route complete: preference has entered the original parameters

Mugi A corgi with a Shiba face

After fine-tuning, generation uses the updated mixture weight even without preferred data or a scoring helper. The model’s $p_\theta$ has changed. Adapting to another dataset requires more parameter updates.

This toy uses density likelihood directly; the earlier diffusion example used noise regression on preferred data. Their common boundary is continued learning on new data that changes the original $\theta$ .

Steering: preserve the model and change this generation

Keep the model and bring in a helper

Nori A polar bear in an orca hat

“Keep the model that already draws dogs. For this red-hat request, find a helper that can recognize red hats. It can help select results; if it also understands unfinished images, perhaps it can offer suggestions along the way.”

Nori wants this request to affect the generation or selection computation while preserving the generative model’s parameters $\theta$ . This is test-time steering. The helper may have its own preparation and training cost, but using it here does not overwrite the original model.

Steering includes intermediate adjustments and final selection. Adding an extra direction at each step is also often called guidance. Today the signal concerns red hats; tomorrow it could concern another task, while the original $\theta$ stays fixed.

Can selecting only at the end still steer?

Nori A polar bear in an orca hat

“Back to the red-hat dog: if the helper only understands finished photos, could we generate several, then keep the highest-scoring one?”

Yes. Independently generate $N$ candidates from the original model, score them, and return only the highest-scoring one. This is Best-of- $N$ . Each candidate’s generation is unchanged, but final selection changes the distribution of returned results, so it is still steering.

It needs no reward gradient or model-internal changes. But if the model rarely draws red-hat dogs, how much generation computation will selection require?

Use $R(x_0)=x_0$ , a preference for larger coordinates. The original model stays equally weighted, and selection occurs after candidates are complete. Inspect whether this scoring rule also preserves the within-class variation we want.

Steering: how do more candidates change the returned distribution?

Nori A polar bear in an orca hat

Predict the change from N = 1 to N = 32, then run A/B. The histogram contains each batch’s returned sample. Inspect the right-class share, peak location, and cost in complete candidates.

Data: Candidates from the fixed model
Input c: This model has no c input
Model parameters θ: Centers, spread, and weights all fixed
Return rule: Keep the best reward from N candidates

Predict, then run a controlled comparison

With the model fixed and R(x₀) = x₀, does increasing N from 1 to 32 only favor the right class, or also change locations and shape within it?

Original mixture Current model / return density Generated histogram

Original model: equal mixture · μ = −2, 2 · σ = 0.55

Compare the curves, then train or generate and inspect where the histogram falls.

Explanation of the standard A/B comparison

Both observations draw candidates from the same model with right weight 0.5. N = 32 adds candidates and selection, not training. Taking maxima favors the right tail and changes the peak and shape. It does not guarantee the original right-class shape or that reward represents the entire requirement.

Example: draw eight independent candidates from the fixed equal mixture and return the largest x₀. The right mode and its upper tail become more frequent. Choosing the smallest favors the left, with the original model means and mixture weight unchanged.

After custom control or Python changes, use your actual plot and parameters; this standard comparison is not a record of that custom run.

Edit Python and run the experiment yourself

Change the parameters or the sampling rule. Python loads when you press Run and executes in your browser. Each run starts fresh; your edits are not saved.

The first opening imports the controls above; reopening preserves your edits. Settings are imported, not accumulated trained parameters or samples, so finite-sample plots need not match exactly.

First change only N near the top and compare candidate cost and returned density. Then change direction to reverse the reward. The model weight stays fixed.

Experiment code (Python)

import math
import numpy as np
import matplotlib.pyplot as plt

# 1. DATA AND SETTINGS -- edit one setting above the model code.
N = 8
direction = 1  # EDIT: R(x0) = direction * x0
rng = np.random.default_rng(42)
sigma = 0.55
grid = np.linspace(-5, 5, 500)
def pdf(x, mu):
    return np.exp(-0.5 * ((x - mu) / sigma) ** 2) / (sigma * np.sqrt(2 * np.pi))
def sample_mixture(n, right=0.5):
    mu = np.where(rng.random(n) < right, 2.0, -2.0)
    return rng.normal(mu, sigma)
base = 0.5 * pdf(grid, -2) + 0.5 * pdf(grid, 2)

# 2. FIXED MODEL -- no training or parameter update.
# 3. GENERATION AND SELECTION -- create actual candidates, then compare rewards.
batch = sample_mixture(1000 * N).reshape(1000, N)
indices = np.argmax(direction * batch, axis=1)
samples = batch[np.arange(1000), indices]
def cdf(x, mu):
    return np.array([0.5 * (1 + math.erf((v - mu) / (sigma * np.sqrt(2)))) for v in x])
F = 0.5 * cdf(grid, -2) + 0.5 * cdf(grid, 2)
after = N * base * (F if direction > 0 else 1 - F) ** (N - 1)
print("Base right-mode probability is still 0.5; complete generations:", 1000 * N)

# 4. PLOTTING -- density curves and the actual returned samples.
plt.figure(figsize=(7, 3.8))
plt.plot(grid, base, color="gray", linestyle="--", label="original base")
plt.plot(grid, after, color="darkorange", label="model / returned density")
plt.hist(samples, bins=np.linspace(-5, 5, 41), density=True, alpha=0.25, color="darkorange", label="1000 generated outputs")
plt.xlabel("x0")
plt.ylabel("density")
plt.xlim(-5, 5)
plt.legend(fontsize=8)
plt.tight_layout()
plt.show()

import math
import numpy as np
import matplotlib.pyplot as plt

# 1. DATA AND SETTINGS -- edit one setting above the model code.
N = 8
direction = 1  # EDIT: R(x0) = direction * x0
rng = np.random.default_rng(42)
sigma = 0.55
grid = np.linspace(-5, 5, 500)
def pdf(x, mu):
    return np.exp(-0.5 * ((x - mu) / sigma) ** 2) / (sigma * np.sqrt(2 * np.pi))
def sample_mixture(n, right=0.5):
    mu = np.where(rng.random(n) < right, 2.0, -2.0)
    return rng.normal(mu, sigma)
base = 0.5 * pdf(grid, -2) + 0.5 * pdf(grid, 2)

# 2. FIXED MODEL -- no training or parameter update.
# 3. GENERATION AND SELECTION -- create actual candidates, then compare rewards.
batch = sample_mixture(1000 * N).reshape(1000, N)
indices = np.argmax(direction * batch, axis=1)
samples = batch[np.arange(1000), indices]
def cdf(x, mu):
    return np.array([0.5 * (1 + math.erf((v - mu) / (sigma * np.sqrt(2)))) for v in x])
F = 0.5 * cdf(grid, -2) + 0.5 * cdf(grid, 2)
after = N * base * (F if direction > 0 else 1 - F) ** (N - 1)
print("Base right-mode probability is still 0.5; complete generations:", 1000 * N)

# 4. PLOTTING -- density curves and the actual returned samples.
plt.figure(figsize=(7, 3.8))
plt.plot(grid, base, color="gray", linestyle="--", label="original base")
plt.plot(grid, after, color="darkorange", label="model / returned density")
plt.hist(samples, bins=np.linspace(-5, 5, 41), density=True, alpha=0.25, color="darkorange", label="1000 generated outputs")
plt.xlabel("x0")
plt.ylabel("density")
plt.xlim(-5, 5)
plt.legend(fontsize=8)
plt.tight_layout()
plt.show()

Package messages and warnings

Does the shifted peak mean the original model learned?

Nori A polar bear in an orca hat

A and B both use a base right weight of $0.5$ , yet the returned peak for $N=32$ sits beyond the original right center. Which interpretation fits both the parameters and plot?

Candidates still come from the same model; taking a maximum favors the right tail and changes the returned density.

Both the parameter record and candidate model are fixed. A shifted returned peak does not establish training.

The plotted peak has shifted toward a tail. Selecting high scores does not automatically preserve the original right-class shape.

Write the selection operation explicitly

Nori A polar bear in an orca hat

Each batch independently generated $x_0^{(1)},\ldots,x_0^{(N)}\sim p_{\mathrm{base}}$ , then computed

i^*=\arg\max_{i=1,\ldots,N}R(x_0^{(i)}),\qquad x_{\mathrm{return}}=x_0^{(i^*)}.

For $R(x_0)=x_0$ , this selects the maximum; $-x_0$ selects the minimum. The code repeats this for $1,000$ batches, returning one per batch. The histogram contains $1,000$ outputs, but the cost is $1,000N$ complete candidates. The original centers, spread, and weight never update.

Why can the returned density differ from the candidate density? Start with the event that a maximum stays below a chosen location.

Why does choosing a maximum concentrate on the tail?

Nori A polar bear in an orca hat

Let $F(x)=P(X\le x)$ be the base cumulative distribution and $f(x)$ its density. The returned maximum is at most $x$ exactly when all $N$ candidates are at most $x$ . Independence gives

P(X_{\max}\le x)=F(x)^N,\qquad f_{\max}(x)=Nf(x)F(x)^{N-1}.

The plot numerically evaluates this density and compares it with actual generated outputs. Increasing $N$ changes the returned distribution and can concentrate it in the tail, which differs from simply drawing from one original component.

Can an extra helper preserve the original model?

Nori A polar bear in an orca hat

Nori finds a network that already recognizes red hats. During generation, both networks stay fixed, and its scores only select results. Does this still count as test-time steering with fixed generative parameters?

Which direction could Nori use?

Nori A polar bear in an orca hat

To preserve $\theta$ , change the sampler’s computation for this generation. For example, a score-based sampler uses $s_\theta(x_t,t)$ , an estimate of the gradient of log image density with respect to the current image $x_t$ .

Suppose this request supplies an additional direction $g_t(x_t)$ . Sampling can temporarily use

\begin{gathered} \widetilde s_t(x_t)\\ =s_\theta(x_t,t)+g_t(x_t),\\ \theta\ \text{fixed}. \end{gathered}

The first term retains the model’s information; the second aims to make results better match the request. This only describes a possible direction. We still need to determine how to obtain $g_t$ and whether adding it corresponds to the distribution we want. Naming the request inside a formula does not solve those questions.

The original network does not learn again; the sampler uses a combined prediction this time. The image $x_t$ keeps changing while the generative parameters $\theta$ can stay fixed.

What should the helper answer for a different request?

Nori A polar bear in an orca hat

“A helper can recognize red hats. But what if the user gives us a blurry photo to restore, or just prefers a particular composition? Is the helper still answering the same question?”

These requests provide different information. A condition specifies desired content, such as “a dog in a red hat.” An observation is an acquired measurement, such as the blurry photo to restore. A reward is a scoring criterion that tells us which results better match a preference.

The blurry measurement can check whether generated results agree with what was observed; a scoring program can compare compositions. The original model does not automatically know this new information. It must enter generation or selection to affect the result.

Does a new obstacle enter the computation by itself?

Nori A polar bear in an orca hat

Two warehouse views with identical start, goal, and blue route. The right view adds a red obstacle; the old route still passes through it. — The blue line is the robot’s route through the warehouse. Diffusion generates that entire route; physical travel time and diffusion time are different.
Select the image to enlarge

When the route model was trained, the aisle was empty. Today a box blocks the original route. We supply neither the new scene nor a collision score, and only double the sampling steps. Does that introduce the box’s location into generation as new information?

How often do both attempts miss?

Nori A polar bear in an orca hat

Suppose each independent base-model generation has probability $p$ of producing a matching red-hat dog. One attempt misses with probability $1-p$ .

Now generate two images. What is the probability that both miss? Then extend to $N$ : compute the probability of all failures and subtract it from $1$ to obtain the probability of at least one hit.

You can also think first, then read the explanation. Your writing stays on this page and disappears when you reload.

More candidates change both success and cost

Nori A polar bear in an orca hat

Independence lets us multiply the miss probabilities. All $N$ candidates miss with probability $(1-p)^N$ ; at least one hit is the complementary event:

\begin{gathered} P(\text{at least one hit})\\ =1-(1-p)^N. \end{gathered}

For $p=0.01$ , $10$ candidates give about a $9.6\%$ chance of at least one hit, and $100$ give about $63.4\%$ . Even a hundred candidates do not guarantee a hit.

Fix a desired success probability $0<a<1$ , with $0<p<1$ . We need $(1-p)^N\le 1-a$ , hence

N\ge\frac{\log(1-a)}{\log(1-p)}.

For small $p$ , $\log(1-p)\approx-p$ , so the required $N$ scales like $1/p$ . Success becomes more likely at the cost of generating more complete candidates. The helper only scores at the end; it has not redirected nearly successful images along the way.

Route complete: changed return rule and generation computation

Nori A polar bear in an orca hat

We generated actual candidates from the same base, scored them, and selected results without updating generative parameters. The changed curve describes returned outputs. More complete candidates cost more generation computation.

Steering can also add intermediate directions; we located that computation above, and the next unit asks how a helper reads noisy images. Here we established that fixed $\theta$ can still let a task signal influence outputs, while higher scores need not produce the user’s complete desired distribution.

If the model stays fixed, why does the image change?

Nori A polar bear in an orca hat

All three proposals are worth trying. Lumi teaches the model to read descriptions, Mugi continues training on preferred photos, and Nori brings a helper into generation or selection.

But the model keeps turning noise into an image. How can Nori say “the parameters stay fixed”? The learned parameters $\theta$ and the changing image $x_t$ are different objects. Changing how we generate this $x_t$ does not necessarily update $\theta$ .

You have now completed each route and its plotted implementation. Use the table to review when requests enter, whether parameters change, and what computation generation uses.

Final review: place the operations under each method

Nori A polar bear in an orca hat

Review the operations you have tried. You can return to the method list above to reread any complete route.

Lumi · When the request enters Give the network images and descriptions during training.

Mugi · When the request enters Start with a model, then select preferred images for further training.

Nori · When the request enters Use a helper’s judgment during generation or selection.

Lumi · Parameter updates Yes: update parameters using described training examples.

Mugi · Parameter updates Yes: update the original parameters on preferred data.

Nori · Parameter updates No: keep the original parameters and adjust generation or selection.

Lumi · Generation Give a description to the model that learned to read it.

Mugi · Generation Use the model already updated on preferred images.

Nori · Generation Use the helper’s signal to adjust sampling or select results.

Both Mugi and Nori receive the request after a model exists. Timing alone is insufficient; next, inspect the parameters.

Lumi and Mugi both change θ through training. Nori preserves θ but may add generation and evaluation computation.

Learning text inputs, writing preferences into parameters, and bringing in a helper are different ways to prepare and use a model.

Distinguish the approaches by when the request enters, whether θ changes, and what generation uses. No proposal is better simply because of who suggested it; the choice depends on the task’s conditions.

New requirement: specify a class on every future call

Lumi A winged deer

We can prepare labelled training data and want the deployed model to accept $c$ directly for left or right outputs, without extra scoring or candidate selection. Which proposal establishes that capability?

Labelled training teaches the model to use c, which can then be changed at deployment.

Fine-tuning an unconditional model on one preferred dataset changes its overall distribution without adding a switchable c input.

This adds generation-time scoring and selection, unlike directly supplying c to the deployed model.

Another requirement: retain a new preference in the original model

Mugi A corgi with a Shiba face

We may update the original $\theta$ , have only newly collected preferred data, and want to keep the model architecture. Deployment supplies no descriptions and performs no scoring or selection. Which operation fits?

This prepares paired data and adds an input rather than updating the original architecture directly on the available preferred data.

Training writes the preference into θ, so deployment uses the updated model directly.

Final selection keeps θ fixed and still needs deployment-time scoring and selection.

Now add this task’s constraint

Nori A polar bear in an orca hat

The user wants to try red hats today and scarves tomorrow, while keeping the original generative model’s parameters fixed. Which proposal meets this constraint?

Lumi makes the condition part of training, which changes learned parameters. Here we want to preserve the original θ.

Although Mugi receives the request later, preferred-data training still updates the original model.

Extra evaluation or sampler adjustments do not necessarily update the generative parameters.

If the model already reads descriptions, must a new prompt retrain it?

Lumi A winged deer

Change the premise: this model already learned a text input from paired data. Today we supply a new description combination within its capabilities and generate with the same $\theta$ , without external scoring or extra guidance. Which account fits?

An established conditional input can be reused during generation. A new description combination is not a new, unlearned input type.

Conditional training establishes the ability to use c. It does not update θ every time a different c is supplied.

The model already accepts c, and the task adds no scoring or guidance. Fixed θ alone does not establish Best-of-N.

Can one workflow combine two methods?

Mugi A corgi with a Shiba face

In the morning, update the original model on preferred data. In the afternoon, fix those updated parameters, generate $16$ candidates per request, and return the highest reward. Classify the operations in their actual order.

The morning data updates θ: fine-tuning. The afternoon keeps that θ fixed and changes the return rule: Best-of-N steering.

The afternoon parameters are fixed, but the morning explicitly updated the original model.

Afternoon selection adds no parameter update. More candidates do not turn selection into fine-tuning.

The requirement is more than a high score: cite the plot

Nori A polar bear in an orca hat

The user now says: “Return the right class more often, while preserving its original within-class distribution shape.” You can use only the earlier $R(x_0)=x_0$ . Someone proposes continually increasing $N$ .

Recall the $N=1$ and $N=32$ comparison. Would you accept that proposal immediately? Cite one feature of the plot and identify an assumption that needs checking. You can revisit the steering route above.

You can also think first, then read the explanation. Your writing stays on this page and disappears when you reload.

Check reward and the requested distribution separately

Nori A polar bear in an orca hat

Do not accept it merely because right-class returns increase. At $N=32$ , the returned peak moves into the original right tail and the within-class shape changes, while the generative model’s right weight stays $0.5$ . This is the distribution of selected maxima, not direct sampling from the unchanged right component.

Recheck whether $R(x_0)=x_0$ represents the full requirement and which distribution the selection rule produces. More candidates can increase maximum reward without guaranteeing the requested within-class shape. A target distribution must be designed and checked; a higher average score alone is insufficient evidence.

Both select photos. What happens afterward?

Mugi A corgi with a Shiba face

Mugi and Nori may both generate a batch of red-hat photos. Which operation is final selection with the original parameters fixed?

Both involve selecting photos. The distinction is whether they are returned directly or used to update the model.

Continuing training on selected images updates the original parameters. That is fine-tuning.

Independent extra candidates add generation and evaluation cost, not a parameter update.

If the helper likes it, is the image good?

Lumi A winged deer

The red-hat scorer loves large patches of red: a red pattern that does not resemble a dog scores highly. Another batch repeats the same dog and pose. Is a higher average reward enough to evaluate steering?

High scores can coexist with implausible images and excessive concentration. Check data fidelity and diversity too.

The red pattern demonstrates that the program can score an implausible input highly.

Even a plausible repeated image loses the different results the user may want.

Could we offer suggestions along the way?

Nori A polar bear in an orca hat

We can now distinguish the proposals: learn descriptions during training, fine-tune on new data, or preserve parameters and use this task’s signal during generation and selection. A late request can still enter; the key is which computation makes it affect results.

Final selection works, but rare targets require many complete candidates, and nearly successful images receive no intermediate correction. Could we tell whether a blurry outline is heading toward a red-hat dog or a corgi before it is finished? Next, we will examine what the helper can see and how to obtain a direction.

Read the same concepts, losses, and comparisons continuously in U7.0 The Model Is Trained. Can We Still Change What We Want?.