Move from preferred-photo proportions to continuous sampling. Plot posteriors, mean estimates, and h_t to inspect uncertainty lost by plug-in.
Prefer continuous reading? Skill manual Read the note · U7.2 →
After seeing corgis in snowy mountains, the user says, “These are all correct, but I prefer this lighting and composition. Could I have several different results?” They give us an aesthetic scoring program: input a finished image and receive a score.
Write a clean, finished image as and its reward as . Higher scores better match this criterion. A reward can simply be an aesthetic score: it need not be a probability or differentiable. First specify how we want to sample; later, check the conditions for computing a direction. We have no noisy-image classifier for this preference, and the model has not learned it. Before changing the sampler, how should the score affect returned results?
“If the program scores images, let us draw many and keep the highest-scoring one. When the user returns tomorrow, give them that same image?”
This uses the score and can find a high-scoring result. But the user also wanted several different results. Does keeping only the highest-scoring image express that request?
“If we want to avoid returning the same image every time, sample as before. More draws will give different results.”
Nori preserves the original variation. But the user specifically preferred certain lighting and composition. If frequencies stay exactly the same, has this score affected the results?
“Could preferred photos appear more often without taking every chance away from other plausible photos? Across several draws, perhaps we would see the preferred style more often, but occasionally another composition.”
Lumi wants to change the proportions in which images are sampled. We want more than one highest-scoring image and more than the unchanged base proportions. What objective would express this idea?
The user wants preferred results more often while retaining different plausible answers. Which approach better expresses both requirements?
It can find a high-scoring result, but repeating one image does not express the request for varied answers.
Making preferred results more frequent while retaining alternatives means choosing a new sampling distribution.
The original proportions retain variation, but the newly supplied scores have no effect.
“What if the scorer especially likes a bright white blob that is not a corgi? Do we want it to appear very often?”
We also want the original data to matter. Let denote the base distribution: probabilities for discrete results, densities for continuous images. It tells us where the original data places more weight. It is not another aesthetic score, and it does not guarantee every high-density result is good.
To combine original weights and reward in one objective, start with . Log turns multiplication into addition; we will see how this connects to KL between distributions. With finite rewards, reweighting alone cannot create results outside the original support.
Switch briefly to training. We have independently drawn data and change model parameters . Here indexes data, not diffusion time. Likelihood and log-likelihood are
Log is strictly increasing, so taking the log of the same likelihood leaves its maximizing parameters unchanged. Replacing products of small numbers with sums also helps computation. Dividing by fixed preserves the maximizer too. Using the data average to estimate a population expectation gives . When the relevant quantities are defined,
The data distribution is fixed, so maximizing average log-likelihood minimizes KL in this direction. Training fixes the data and changes the model. We will instead fix the base and choose a new sampling distribution . Keep these settings distinct.
Does “log preserves order” let us swap log and averaging? Suppose data probabilities for are , while the model assigns . Average probability is , favoring . Average log probability is
For , its derivative is , zero at . The second derivative is negative, so this is the maximum. It retains the rarer . Taking a log after averaging and averaging logs optimize different objectives.
Both are plausible, with different lighting and poses. The example assumes the program prefers B. These probabilities and rewards are given toy values, not model measurements.
Select the image to enlargeSimplify the possible results to , originally equally likely, with rewards . The new distribution we choose, , determines their future frequencies.
To favor reward while retaining the data’s influence, score each image by . Here sets reward influence, and is the base. These are two discrete results, each with probability ; continuous images would use density.
Imagine repeatedly drawing from and averaging this score. Will maximizing that average keep both photos, or choose every time? Predict the result and explain why.
You can also think first, then read the explanation. Your writing stays on this page and disappears when you reload.
Both log base probabilities equal ; only the reward differs. The new probabilities sum to , so the average score is
For any , assigning more probability to raises the average, eventually returning every time. Both images were plausible, but this average score never requires retaining the other answer.
For finitely many candidates, write each fixed image score as
The average cannot exceed the largest . Assigning all mass to a highest-scoring candidate attains that value. Ties allow mixing among maximizers, but still do not encourage retaining other answers.
Continuous spaces need more care: an ordinary density may not attain a point-concentrated maximum; increasingly concentrated sequences might only approach it. The shared problem is averaging fixed per-image scores without separately measuring the new distribution’s concentration.
The objective included both reward and log base probability, yet chose only . Which original requirement still needs to enter?
The original objective only averages per-image scores; it does not measure excessive concentration of the whole distribution.
Reward already appears. Increasing it only favors B more here; it adds no requirement to retain alternatives.
The equal base probabilities already enter. What is missing is a measure of the new distribution’s concentration.
Entropy measures how spread out a distribution is: . For our two photos, always choosing gives , while equal probabilities give . Adding entropy makes spreading probability across answers contribute too:
Every expectation averages over drawn from the new . The scoring program, base, and stay fixed. We change the frequencies of results.
Expand entropy. All three terms average over the same , so the objective value becomes
Use and the product and quotient rules for log. How would you write the three terms inside as one log ratio? Start with its numerator and denominator.
You can also think first, then read the explanation. Your writing stays on this page and disappears when you reload.
The added logs belong in the numerator, and the subtracted log in the denominator:
Look at the numerator: the original weight is multiplied by , giving higher-reward results larger multipliers. This resembles the desired reweighting, but is it already a probability distribution?
Keep base probabilities and rewards . With , the multipliers are , giving weights for and for . How should we interpret them?
Relative weights may exceed 1; probabilities may not. Normalization restores total mass to 1.
B’s value exceeds 1 and the total is 2, so these cannot yet be probabilities.
A relative weight exceeding 1 does not invalidate the multiplier. Normalize rather than clip it.
Write the sum or integral of weights as . Dividing by it defines a candidate distribution:
For two photos, is a sum; for continuous images, it integrates over all . Assume and that relevant expectations are defined. We still need to check that this candidate maximizes our objective.
Return from A/B outcomes to the continuous mixture from U7.0. Reward can favor the right or two regions. Inspect actual samples before proving why this candidate optimizes the objective.
Predict, then inspect the distribution
Starting from an equal mixture, does reward weighting change only the mode shares, or also shape and location within each mode?
A has λ = 0; B uses your reward and λ. Draw 1,000 samples from each one-dimensional numerical CDF.
The gray curve shows the original mixture. Running adds actual samples; a reference curve does not establish that it was sampled.
Explain using the plot and numbers
Weighting acts on every x0 and can change within-mode shape. The right reward favors the right mode and its tail; the symmetric reward can retain two peaks. A grid makes normalization and sampling possible here; images need the later sampling methods.
This explains the standard comparison. After changing controls or Python, use the settings and plots you actually ran.
Fixed base: ½N(-2,0.55²) + ½N(2,0.55²). Normalize p_base exp(λR) on [-6,6] with Δx = 0.025, invert the CDF, and sample uniformly within a bin. This is a numerical one-dimensional approximation, not a diffusion network.
Change the parameters or the sampling rule. Python loads when you press Run and executes in your browser. Each run starts fresh; your edits are not saved.
The first opening imports the controls above; reopening preserves your edits. Settings are imported, not accumulated trained parameters or samples, so finite-sample plots need not match exactly.
The code implements this method and plots its distribution with Matplotlib. The first opening imports current controls; every run samples afresh. Change one parameter at the top. The native plot is not a record of edited Python output.
import math
import numpy as np
import matplotlib.pyplot as plt
# 1. SETTINGS -- change one, run again, and explain the plotted evidence.
strength = 2
reward_kind = "right" # "right" or "two"
rng = np.random.default_rng(42)
sigma = 0.55
grid = np.linspace(-6, 6, 481)
dx = grid[1] - grid[0]
# 2. FIXED MODEL -- analytic functions, no network training in this cell.
def pdf(x, mu=0.0, var=1.0):
return np.exp(-0.5 * (x-mu)**2 / var) / np.sqrt(2*np.pi*var)
def R(x):
return x if reward_kind == "right" else -(np.abs(x)-2.5)**2
base = 0.5*pdf(grid, -2, sigma**2)+0.5*pdf(grid, 2, sigma**2)
target = base*np.exp(strength*R(grid))
target /= target.sum()*dx # finite-grid quadrature reference
reference = target
reference_label = "Tilt reference (quadrature on [-6,6])"
extra_figure = None
# 3. GENERATION -- directly invert the finite-grid target CDF.
# This is possible in one dimension; it does not solve high-dimensional sampling.
cdf = np.cumsum(target*dx)
indices = np.searchsorted(cdf, rng.random(1000)).clip(0, len(grid)-1)
samples = grid[indices]+rng.uniform(-dx/2, dx/2, 1000)
print("Reference mass:", target.sum()*dx)
print("Target mean reward:", np.sum(target*R(grid))*dx)
# 4. PLOT AND CHECK -- the histogram is from the samples above.
fig, ax = plt.subplots(figsize=(8, 3.6))
ax.plot(grid, base, "--", color="gray", label="Original base")
ax.plot(grid, reference, ":", color="black", label=reference_label)
ax.hist(samples, bins=np.linspace(-6,6,49), weights=np.ones(len(samples))/(len(samples)*0.25), alpha=0.5, label="Actual generated samples")
ax.set(xlabel="clean x0", ylabel="density", xlim=(-6,6))
ax.legend(fontsize=8)
fig.tight_layout()
print("Outputs:", len(samples), "mean:", np.mean(samples), "right share:", np.mean(samples > 0))
print("Outside the plotted window:", np.sum(np.abs(samples)>6))
plt.show()Substitute into the log ratio. Separate the fixed . The remaining average of is negative KL:
KL measures the difference between these distributions. It is nonnegative and zero when they coincide. With base, reward, and fixed, which maximizes the objective?
Subtracting a nonnegative quantity gives a maximum when that quantity is zero. We choose the new distribution, not the fixed scorer or base.
The objective subtracts KL. Larger KL lowers the value.
Fixed base, reward, and strength already determine Z_R. Choosing a new distribution does not change it.
“We now have new proportions that favor the user’s style while keeping the original distribution involved. Can ‘do not stray too far from the base’ appear directly in the objective?”
Yes. Entropy plus expected log base equals negative KL. Divide the objective by positive , which preserves its maximizer:
To see why, expand the KL definition:
This KL direction differs from the earlier training example: now the expectation is over the we choose.
A larger makes the penalty coefficient for departing from the base smaller. We allow preference more influence, but cannot conclude that images become more plausible.
We could maximize subject to , where is the allowed departure.
With positive multiplier , the Lagrangian is
The final is constant in . For a fixed multiplier, this has the same optimizer as the KL-penalized objective. But arbitrary choices of do not make the constrained and penalized problems equivalent.
For example, if has finite KL, choose . Optimality of implies that any feasible satisfies
Thus also solves the constrained problem for this matched . This is the reason the two forms connect.
Suppose we observe , such as an actually received blurry photo. Let describe how likely an original image is to produce this observation, and choose and . Then
When , normalization gives the Bayes posterior . Here is an observation, unlike the desired condition “corgi.” It also differs from the base posterior given a sampling-time below. In this example, the observation enters as a log-likelihood reward.
Fix a prompt, view as an answer, and let be the original answer distribution. A common preference objective is
Here controls retention of the reference. For this ideal distribution-level problem, and optimal relative weights are . This has the same structure as KL regularization in RLHF.
Actual language-model training usually updates policy parameters and faces capacity and optimization limits; some full objectives also include pretraining terms. This ideal optimal distribution is therefore not the exact outcome of every RLHF algorithm. Here we instead preserve generative parameters and try to change sampling.
Return to the two plausible photos. initially have equal probability and rewards . At , the unnormalized weights are and , totaling .
We have specified the desired proportions. Will your return rule actually produce them? Predict the share of , then choose original sampling, reward maximization, or sampling the new weights. After one run, change the rule or the original proportions and compare.
A worked example to compare
A and B initially have equal probability, with rewards 0 and 1. At λ = log 3, their weights are 1/2 and 3/2, giving target proportions A: 25%, B: 75%. Original sampling stays at 50/50; always maximizing reward returns only B. Sampling the new weights gives the target in this two-outcome example.
This is a discrete A/B model with direct weighted sampling. Reward maximization chooses from the support; it is separate from finite-candidate Best-of-N.
Change the parameters or the sampling rule. Python loads when you press Run and executes in your browser. Each run starts fresh; your edits are not saved.
The first opening imports the controls above; reopening preserves your edits. Settings are imported, not accumulated trained parameters or samples, so finite-sample plots need not match exactly.
import math
import random
base_b = 0.5
strength = 1.0986122886681096
rule = "tilt" # "base", "best", or "tilt"
trials = 1000
rng = random.Random(42)
# Reward(A) = 0; Reward(B) = 1.
weights = [1 - base_b, base_b * math.exp(strength)]
target_b = weights[1] / sum(weights)
if rule == "base":
return_b = base_b
elif rule == "best":
return_b = float(base_b > 0)
else:
return_b = target_b
samples = rng.choices(["A", "B"], weights=[1 - return_b, return_b], k=trials)
print(f"Target B: {target_b:.1%}")
print(f"Rule B: {return_b:.1%}")
print(f"Observed B: {samples.count('B') / trials:.1%}")The new proportions use both base and reward. If the photos originally had unequal probabilities, multiply each original weight by its own reward multiplier rather than comparing rewards alone. Large multipliers can greatly amplify low-density regions; including a base does not guarantee every result is plausible.
We use ideal to define the target. Actual generation uses a learned model, whose errors still matter. Now return to intermediate guidance: the scorer understands finished images, while we only have a noisy image.
First assume base and reward are differentiable at the clean end. Take logs of and differentiate with respect to the clean image:
“Original direction plus reward direction” looks much like U7.1. Indeed, with recovers the clean conditional score.
But we do not have finished , only . Does replacing subscript with let the same reward evaluate noise? First consider which clean photos could have produced this blurry image.
Fix the same . Equal posterior probabilities for A and B are a separate given assumption; equal base probabilities earlier do not imply it.
Select the image to enlargeThis could have come from or , whose rewards differ. What would you do first to bring clean-image reweighting to this state?
Each source has its own probability and reward weight. Uncertainty does not permit choosing just one source in advance.
The original model supplies base information; the new scorer has not entered the computation automatically. Ignoring it does not produce the new target.
The aesthetic program was meant to judge composition and lighting in finished photos. Now it receives blurry noise. Even if it returns a number, that need not measure the intended request. Defining on clean does not make it valid on .
The same blurry outline could come from different photos with different rewards. Connecting the clean target to intermediate states requires handling those sources.
To find the noisy distribution corresponding to , draw a clean image from it and apply the original forward process. Use conditional density for the noisy image given a clean source. Sum all sources’ contributions:
We integrate over possible clean sources , holding the observed fixed. After substituting , we can rewrite the contributions as source probabilities given the current image.
Substituting gives inside the integral. This is the base joint density, which Bayes’ rule also writes as
When is fixed and we integrate over sources , which factor can move outside?
The integration variable is the clean source; base density at the fixed noisy image does not depend on it.
Different sources have different probabilities. This posterior supplies their weights and cannot leave the integral as a whole.
A and B have different reward multipliers. This factor depends on the clean source being integrated.
After taking out , name the remaining average weight :
Read the expectation as: “Given this , which clean photo could have produced it? Compute each source’s reward multiplier, then average by its probability.” This uses the base posterior , not a reward-modified posterior or an average over all data.
The two sources are equally likely under the base posterior, with rewards . Again let , giving multipliers . How should we compute for this ?
h_t is the conditional average of reward multipliers, retaining both sources’ contributions.
The scorer prefers , but this need not have come from . The base posterior still gives each source probability one half. Taking only the largest multiplier removes the other source’s contribution.
We need the average weight over possible sources, not the score of the best possible source.
Averaging rewards first gives and then the exponential multiplier . Computing both multipliers first and averaging gives .
Exponentiation is nonlinear, so these orders need not agree. The definition of computes each source’s multiplier first, then takes the posterior average. Keep that order.
“Including all possible sources sounds hard. We already have a denoiser. Could we denoise the current image, then give that to the aesthetic program?”
Recall . Subtract predicted noise and divide out the signal scale:
Here and . Under ideal squared-error noise estimation, this equals the base posterior mean ; actual networks also have estimation error. It supplies a clean estimate, not a uniquely recovered true source.
Mugi proposes using instead of the needed . Which computation would this save, and which information might it remove?
Recall : ideally it already averages possible sources. Mugi wants to score only this estimate, using instead of averaging individual source multipliers in . Does one mean estimate retain the average weight of all possible sources? Think about the order of averaging and nonlinear operations.
You can also think first, then read the explanation. Your writing stays on this page and disappears when you reload.
Temporarily replace high-dimensional images with numbers: is and is , still equally likely given . Choose and , reproducing the previous rewards and multipliers.
Average the data first to get , then evaluate its multiplier:
Computing source multipliers first and averaging gave . Even a linear reward leaves an outer exponential that makes the orders differ. Real image rewards are usually nonlinear too, so equality is even less justified.
Replacing the source average with a single denoised prediction is a plug-in approximation. It discards source uncertainty; the omitted quantity is more than a normalizing constant.
Fix the noisy observation and inspect possible clean sources and their mean. A second plot shows h_t and plug-in across observations. See what uncertainty is lost before differentiating.
Predict, then inspect the distribution
For a fixed x_t = 0, how do possible clean sources change as noise increases? Does scoring their mean equal averaging their reward multipliers?
A uses noise time t = 0.08; B uses your t. The observation, reward, and λ stay fixed.
The gray curve shows the original mixture. Running adds actual samples; a reference curve does not establish that it was sampled.
Explain using the plot and numbers
The posterior describes possible clean sources. h_t averages their exponential reward multipliers; plug-in scores the mean location. A two-mode posterior can have a mean in a low-density gap. Nonlinearity makes the orders differ. The lower curves are weight functions, not densities or a claimed generated distribution.
This explains the standard comparison. After changing controls or Python, use the settings and plots you actually ran.
Sample 1,000 possible sources from the analytic Gaussian-mixture Bayes posterior. h_t uses analytic Gaussian integrals; plug-in uses the analytic posterior mean. This is the base posterior of a fixed noisy observation, not a reward-tilted posterior.
Change the parameters or the sampling rule. Python loads when you press Run and executes in your browser. Each run starts fresh; your edits are not saved.
The first opening imports the controls above; reopening preserves your edits. Settings are imported, not accumulated trained parameters or samples, so finite-sample plots need not match exactly.
The code implements this method and plots its distribution with Matplotlib. The first opening imports current controls; every run samples afresh. Change one parameter at the top. The native plot is not a record of edited Python output.
import math
import numpy as np
import matplotlib.pyplot as plt
# 1. SETTINGS -- change one, run again, and explain the plotted evidence.
strength = 2
noise_time = 0.35
observed_x = 0
reward_kind = "two" # "right" or "two"
rng = np.random.default_rng(42)
sigma = 0.55
grid = np.linspace(-6, 6, 481)
dx = grid[1] - grid[0]
# 2. FIXED MODEL -- analytic functions, no network training in this cell.
def pdf(x, mu=0.0, var=1.0):
return np.exp(-0.5 * (x-mu)**2 / var) / np.sqrt(2*np.pi*var)
def Phi(x):
return 0.5 * (1 + np.vectorize(math.erf)(np.asarray(x) / np.sqrt(2)))
def R(x):
return x if reward_kind == "right" else -(np.abs(x)-2.5)**2
def moments(t):
a = np.exp(-8*t)
return np.sqrt(a), a*sigma**2 + 1-a
def sigmoid(v):
return np.exp(-np.logaddexp(0, -v))
def post_params(x, t):
signal, var = moments(t)
right = sigmoid(4*signal*x/var)
gain = signal*sigma**2/var
left_mu = -2 + gain*(x + 2*signal)
right_mu = 2 + gain*(x - 2*signal)
variance = sigma**2*(1-np.exp(-8*t))/var
return right, left_mu, right_mu, variance
def posterior_mean(x, t):
w, m0, m1, v = post_params(x, t)
return (1-w)*m0+w*m1
def h(x, t):
if t == 0:
return np.exp(strength*R(x))
w, m0, m1, v = post_params(x, t)
if reward_kind == "right":
return (1-w)*np.exp(strength*m0+0.5*strength**2*v)+w*np.exp(strength*m1+0.5*strength**2*v)
d = 1+2*strength*v
def component(mu):
result = 0.0
for side in [-1, 1]:
center = (mu+2*strength*v*side*2.5)/d
factor = np.exp(-strength*(mu-side*2.5)**2/d)/np.sqrt(d)
result += factor*Phi(side*center/np.sqrt(v/d))
return result
return (1-w)*component(m0)+w*component(m1)
base = 0.5*pdf(grid, -2, sigma**2)+0.5*pdf(grid, 2, sigma**2)
target = base*np.exp(strength*R(grid))
target /= target.sum()*dx # finite-grid quadrature reference
reference = target
reference_label = "Tilt reference (quadrature on [-6,6])"
extra_figure = None
# 3. POSTERIOR -- sample possible clean sources of this fixed noisy observation.
w, m0, m1, v = post_params(observed_x, noise_time)
samples = np.where(rng.random(1000) < w, m1, m0)+np.sqrt(v)*rng.normal(size=1000)
reference = (1-w)*pdf(grid, m0, v)+w*pdf(grid, m1, v)
reference_label = "Base posterior p(x0 | xt)"
mean = posterior_mean(observed_x, noise_time)
exact = h(observed_x, noise_time)
plugin = np.exp(strength*R(mean))
print("Posterior mean:", mean, "h_t:", exact, "plug-in:", plugin)
extra_figure, ax = plt.subplots(figsize=(8, 3.2))
ax.plot(grid, h(grid, noise_time), label="h_t: average the reward MULTIPLIERS")
ax.plot(grid, np.exp(strength*R(posterior_mean(grid, noise_time))), label="plug-in: score the posterior MEAN")
ax.axvline(observed_x, color="gray", linestyle=":")
ax.set(xlabel="noisy observation x_t", ylabel="weight (not density)")
ax.legend(fontsize=8)
extra_figure.tight_layout()
# 4. PLOT AND CHECK -- the histogram is from the samples above.
fig, ax = plt.subplots(figsize=(8, 3.6))
ax.plot(grid, base, "--", color="gray", label="Original base")
ax.plot(grid, reference, ":", color="black", label=reference_label)
ax.hist(samples, bins=np.linspace(-6,6,49), weights=np.ones(len(samples))/(len(samples)*0.25), alpha=0.5, label="Actual generated samples")
ax.set(xlabel="clean x0", ylabel="density", xlim=(-6,6))
ax.legend(fontsize=8)
fig.tight_layout()
print("Outputs:", len(samples), "mean:", np.mean(samples), "right share:", np.mean(samples > 0))
print("Outside the plotted window:", np.sum(np.abs(samples)>6))
plt.show()We have . Follow U7.1: take logs, then differentiate with respect to the current image .
does not change with the image, and the base term gives the original score. Which term supplies the extra direction this time? Try writing the new target’s score.
You can also think first, then read the explanation. Your writing stays on this page and disappears when you reload.
Termwise differentiation gives
The extra direction is the log-gradient of : how the average weight of compatible sources changes with the current image. Exact is hard to compute, so a common approach replaces it with the plug-in estimate.
Assume reward and denoised prediction are differentiable. We use for suggestions while keeping both networks fixed. Which calculation gives a direction for the current image?
Input gradients can pass through a fixed network. Fixed parameters do not make the output’s input derivative zero.
That changes the generative model, violating the fixed-parameter constraint.
Holding the denoised prediction constant removes its dependence on the current image. Its derivative with respect to the current image is also needed.
The plug-in log is , so the extra direction is approximated by
Write out the whole dependency using the chain rule:
The reward gradient describes changes to the clean estimate that increase its score. The Jacobian describes how that estimate changes with the current noisy image. Their product converts the direction back to the we can modify. Fixed still allows this input gradient.
Our example already showed unequal weights, so this approximation cannot simply be called exact tilting. Even exact scores need the correct initial noisy distribution and reverse dynamics for actual sampling. Network estimation, finite noise, and a finite-step sampler add approximations to check.
The same reweighting relation holds at starting time :
If the forward process has completely forgotten the clean source, and are independent, so
Then , allowing the same base start. Finite noise need not achieve full independence. Even highly accurate scores do not establish exact sampling when started from a mismatched distribution.
A common practical route converts noisy images into denoised predictions, then uses an external differentiable request for a direction. Universal Guidance discusses such use. Separating it from the exact posterior average shows where source uncertainty was omitted.
For example, Calibrated Test-Time Guidance analyzes how common approximations bias Bayesian posterior sampling and proposes a consistent estimator under its conditions. Are We Really Tilting? separates mode selection from excessive reward pursuit within modes in Gaussian and mixture examples, and studies reward damping. Read each result with its assumptions; do not extend it to arbitrary rewards and samplers.
Methods can improve posterior approximations, adjust guidance strength, or constrain updates. But a higher average reward checks only one quantity. Excessive preference strength may suppress plausible lower-scoring modes. Assess reward, plausibility, diversity, and whether the defined target is actually sampled separately.
A batch generated with plug-in guidance has higher average reward than before. Does this number alone prove that the sampler draws from our defined ?
Choose a reward and λ from your experiment. Use the target curve, generated histogram, and a posterior-mean or h_t value to explain the evidence needed for higher reward, a correctly defined weight, and actual target sampling.
You can also think first, then read the explanation. Your writing stays on this page and disappears when you reload.
Mean reward addresses the scoring metric. Normalization and derivation define the target. Sampling requires comparing proportions, shape, and missing regions. A one-dimensional finite histogram fluctuates and does not provide the same exact check for real images.
Now let base probabilities of be , keeping rewards and , hence multipliers . Which reweighted proportions meet the target?
A is originally three times as common, while B has three times the reward multiplier. The products are equal.
That ignores the original probabilities and uses only multipliers 1:3. The base proportions have changed.
That retains the original proportions without applying B’s multiplier of three.
For our two plausible photos, we wanted preferred results more often while retaining alternatives and the base’s influence. Reward tilting defines new proportions. Intermediate guidance must consider compatible clean sources. Replacing their average with one denoised estimate eases computation, but remains an approximation.
If the program only returns scores and cannot be differentiated, we cannot directly compute that direction. Could we generate candidates and change returned proportions by comparing scores alone? Continue in U7.3, where we will examine candidate counts and computation cost.
Read the objective, posterior average, and plug-in derivation continuously in U7.2 High Reward and Plausible Data: Which Distribution Should We Sample?.