Learning goal: full position-free attention does not encode token order. Permuting tokens reorders rows and columns of the attention matrix without changing their corresponding values. Other questions concern why logits are divided by √d and when the L²d computation dominates.

Left: token content, with original order in gray. Right: L×L attention matrix, rows as queries and columns as keys.

Views: scaling and entropy, positional-encoding strength, and projection versus pairwise cost.

d = — Mean entropy at these settings = — Uniform entropy log L = — Mean maximum weight = — max|Attn(Px) − P·Attn(x)| = — L²d ÷ L·d² at these settings = —

Observe: shuffle order. The attention matrix's rows and columns reorder together. The equivariance discrepancy is around 10⁻¹⁶, consistent with rounding. This full position-free layer does not identify original indices. Without scaling, increasing d to 1024 gives entropy about 0.10 and maximum weight about 0.96 in the displayed experiment. Scaling keeps attention less concentrated here. The √d factor controls initial logit variance under the stated random-vector assumptions. The positional view measures how this example's logits depend on position: at scale 1, the displayed explained-variation statistic is 4.4%; it exceeds half around scale 3.5. This is an experiment-specific measure: position and content scale both affect the computation. In the cost view, the nominal L²d term exceeds Ld² when L>d; real runtime also depends on constants and implementation.