U3.0 Why Can Straight Reference Paths Produce Curved ODE Trajectories?

Builds on2 notes

Generation repeatedly evaluates the current velocity and applies an Euler update:

x^{(k+1)} = x^{(k)}+\Delta t\, v_\theta\!\left(x^{(k)},t_k\right).

Our linear reference paths have constant velocities. Does their optimal average also remain constant along a generated trajectory? If so, perhaps one large step would suffice.

But consider the question carefully:

Do straight reference paths imply straight Flow ODE trajectories?

Two crossing lines expose the issue

Consider two endpoint pairs:

(-1,-1)\longrightarrow(1,1),\qquad (1,-1)\longrightarrow(-1,1).

Both reference lines cross the origin at $t=1/2$ , with velocities

(2,2),\qquad(-2,2).

At the shared input, squared-error regression outputs their mean:

v_{1/2}(0) =\frac12(2,2)+\frac12(-2,2) =(0,2).

The upward arrow belongs to neither reference path. It averages the two compatible directions.

This one arrow alone does not prove a trajectory bends. It only proves that the marginal velocity differs from each reference velocity there.

Now thicken each endpoint into a small cloud, turning two singular lines into smooth streams. Follow the averaged field from the lower-left stream:

Before overlap, the compatible stream mostly moves rightward and upward.
Within overlap, opposing horizontal signals cancel increasingly.
In the symmetric smooth construction, uniqueness prevents left and right trajectories from meeting and then separating. The left trajectory remains on its side and eventually turns leftward.

Its horizontal velocity therefore changes through

\text{rightward}\longrightarrow0\longrightarrow\text{leftward}.

Changing direction bends the trajectory. The zero-width two-point toy concentrates this behavior at a singular crossing; smoothing spreads the turn over an interval.

Constant reference velocities do not imply a marginal velocity that remains constant along an ODE trajectory.

Switch pairings while keeping both endpoint distributions fixed. Independent pairing creates conflicting directions; position-based pairing makes reference routes more parallel in this toy and can straighten the marginal trajectories.

Changing velocity makes coarse integration difficult

The ODE specifies instantaneous velocity:

\frac{\mathrm dx_t}{\mathrm dt}=v_t(x_t).

The exact displacement over an interval requires integration:

x_{t+h} = x_t+\int_t^{t+h}v_\tau(x_\tau)\,\mathrm d\tau.

A numerical solver approximates this continuous integral with finitely many calculations.

Euler evaluates the network once at the beginning of the interval and uses

x_{t+h}\approx x_t+h\,v_\theta(x_t,t).

A network evaluation is one call to the model. Euler uses one per step; other solvers may use several. This cost strongly affects generation time.

Euler ignores within-step velocity changes. It resembles checking navigation only occasionally: constant direction and speed are easy, while turns and speed changes make a long unchecked movement inaccurate.

Consequently,

Nearly constant velocity can permit accurate coarse steps.
Rapid velocity changes can require a finer grid to control endpoint error.

We now have two directions to investigate:

Can we make generated trajectories closer to constant-speed straight lines, reducing the evaluations needed for accurate sampling?

Or can a solver adapt its step size, taking large steps in stable regions and shorter ones where velocity changes quickly?

The first changes the learned transport geometry; the second changes numerical integration.

Check your understanding

References

Liu, X., Gong, C., Liu, Q. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. ICLR 2023.
Lipman, Y., et al. Flow Matching for Generative Modeling. ICLR 2023.
Tong, A., et al. Improving and Generalizing Flow-Based Generative Models with Minibatch Optimal Transport. TMLR 2024.