The key finding is that success depends on whether the architecture realizes the required inference primitives: belief accumulation (integrating evidence), belief transport (propagating beliefs through dynamics), and random-access binding (retrieving hypotheses by content). Transformers realize all three primitives; Mamba realizes accumulation and transport; LSTMs realize only accumulation of static sufficient statistics; MLPs realize none. This taxonomy explains the empirical pattern: each architecture succeeds precisely on tasks demanding only the primitives it can implement. But how does gradient descent learn to implement these primitives? Why does cross-entropy training produce the geometric structures required for Bayesian inference-orthogonal key bases, progressive query alignment, low-dimensional value manifolds? This paper as Lemma 2 (Mech-anism). Paper I establishes existence: transformers can implement exact Bayesian inference, and different architectures realize different primitives. This paper establishes mechanism: when an architecture supports a given primitive, standard cross-entropy training induces gradient dynamics that reliably construct the corresponding probabilistic computation. Paper III establishes scaling: these mechanisms persist in production LLMs. We analyze a single-head attention block trained with cross-entropy and derive all first-order gradients with respect to scores 𝑠 𝑖 𝑗 , queries 𝑞 𝑖 , keys 𝑘 𝑗 , and values 𝑣 𝑗 . We show that the resulting gradient dynamics implement an implicit EM-like algorithm, and that this EM structure is precisely what enables the inference primitives to emerge.

Our main contributions are:

(1) Complete first-order analysis of attention gradients. We derive closed-form expressions for 𝜕𝐿/𝜕𝑠 𝑖 𝑗 , 𝜕𝐿/𝜕𝑞 𝑖 , 𝜕𝐿/𝜕𝑘 𝑗 , and 𝜕𝐿/𝜕𝑣 𝑗 under cross-entropy loss, in a form that makes their geometric meaning transparent. (2) Advantage-based routing law. We show that score gradients satisfy

where 𝑏 𝑖 𝑗 = 𝑢 ⊤ 𝑖 𝑣 𝑗 is a compatibility term. Since gradient descent subtracts the gradient, scores decrease for positions whose compatibility exceeds the current attention-weighted mean (positive gradient), and increase for positions below average (negative gradient).

(3) Responsibility-weighted value updates and specialization. Values evolve according to

a responsibility-weighted average of upstream gradients. This induces a positive feedback loop: queries route to values that help them; those values move toward their users, reinforcing routing and creating specialization. (4) Two-timescale EM interpretation. We show that these dynamics implement an implicit EM-like algorithm: attention weights act as soft responsibilities (E-step), values as prototypes updated under those responsibilities (M-step), and queries/keys as parameters of the latent assignment model. Attention often stabilizes early, while values continue to refine-a frameprecision dissociation that matches our empirical observations in wind tunnels and large models. ( 5) Toy experiments and EM vs. SGD comparison. In synthetic tasks, including a sticky Markov-chain sequence, we compare an EM-motivated learning rate schedule (with larger LR for values) to standard SGD. The EM-like schedule reaches low loss, high accuracy, and sharp predictive entropy significantly faster; SGD converges to similar solutions but with slower and more diffuse routing. PCA visualizations of value trajectories reveal emergent low-dimensional manifolds.

Taken together with [1], our results provide a unified story: Gradient descent ⇒ Bayesian manifolds ⇒ In-context inference.

Clarification on “Bayesian inference.” Throughout this paper, “Bayesian inference” refers to the Bayesian posterior predictive over latent task variables-not a posterior over network weights. Specifically, we address filtering and hypothesis elimination in tasks whose posteriors factorize sequentially, not general Bayesian model selection. We show that cross-entropy training sculpts geometry that implements these computations over in-context hypotheses.

We analyze a single attention head operating on a sequence of length 𝑇 . Indices 𝑖, 𝑗, 𝑘 run from 1 to 𝑇 unless stated otherwise.

At each position 𝑗 we have an input embedding 𝑥 𝑗 ∈ R 𝑑 𝑥 . Linear projections produce queries, keys, and values:

Attention scores, weights, and context vectors are:

The output of the head is passed through an output projection to logits:

and we train with cross-entropy loss

where 𝑦 𝑖 is the target class at position 𝑖.

For compactness we define:

Here 𝑢 𝑖 is the upstream gradient at position 𝑖, indicating how 𝑔 𝑖 should move to reduce the loss. The scalar 𝑏 𝑖 𝑗 measures compatibility between the error signal 𝑢 𝑖 and value 𝑣 𝑗 , and E 𝛼 𝑖 [𝑏] is the attention-weighted mean compatibility for query 𝑖.

We now derive all relevant gradients without skipping steps, focusing on forms that reveal their geometric meaning.

For each 𝑖, the cross-entropy gradient with respect to logits is

This propagates back to the context 𝑔 𝑖 as

Intuitively, -𝑢 𝑖 is the direction in value space along which moving 𝑔 𝑖 would increase the logit of the correct token and decrease loss.

Since 𝑔 𝑖 = 𝑗 𝛼 𝑖 𝑗 𝑣 𝑗 , we have

Thus

Geometrically, 𝑣 𝑗 is pulled toward an attention-weighted average of upstream error vectors from all queries that use it.

Because 𝑔 𝑖 depends linearly on 𝛼 𝑖 𝑗 , we obtain

The scalar 𝑏 𝑖 𝑗 = 𝑢 ⊤ 𝑖 𝑣 𝑗 measures the instantaneous compatibility between the upstream gradient 𝑢 𝑖 and value vector 𝑣 𝑗 . Since 𝜕𝐿/𝜕𝛼 𝑖 𝑗 = 𝑏 𝑖 𝑗 , increasing 𝛼 𝑖 𝑗 increases the loss when 𝑏 𝑖 𝑗 > 0 and decreases the loss when 𝑏 𝑖 𝑗 < 0. Thus, -𝑏 𝑖 𝑗 represents the immediate marginal benefit (in loss reduction) of allocating additional attention mass to position 𝑗 for query 𝑖.

For fixed 𝑖, the softmax Jacobian is

Applying the chain rule,

Hence

It is convenient to define an advantage quantity with the sign chosen to align with gradient descent:

Under this definition, 𝐴 𝑖 𝑗 > 0 indicates that increasing 𝑠 𝑖 𝑗 (and hence 𝛼 𝑖 𝑗 ) would reduce the loss. Gradient descent therefore increases attention scores toward positions with positive advantage and decreases them otherwise. This is an advantage-style gradient: since gradient descent subtracts the gradient, scores decrease for positions whose compatibility 𝑏 𝑖𝑘 exceeds the attention-weighted average (positive gradient), and increase for positions below average (negative gradient). Since high 𝑏 𝑖𝑘 = 𝑢 ⊤ 𝑖 𝑣 𝑘 indicates 𝑣 𝑘 aligns with the error direction, this reallocates attention away from harmful values toward helpful ones.

Similarly, for fixed 𝑗,

Thus,

Finally, gradients propagate to 𝑊 𝑄 and 𝑊 𝐾 as

For learning rate 𝜂 > 0, gradient descent gives

The remainder of the paper analyzes the coupled dynamics induced by these updates.

We now unpack the implications of the gradient flows in Section 3, focusing on the interaction between routing (via scores and attention) and content (via values).

Equation ( 23) shows that for fixed query 𝑖, attention reallocates mass away from positions whose values are worse-than-average (i.e., 𝑏 𝑖𝑘 > E 𝛼 𝑖 [𝑏]) for 𝑢 𝑖 and toward those that are better-thanaverage. Recall the advantage 𝐴 𝑖 𝑗 := -(𝑏 𝑖 𝑗 -E 𝛼 𝑖 [𝑏]) defined earlier: gradient descent increases attention scores toward positions with positive advantage, implementing an advantage-based routing rule.

Define the attention-weighted upstream signal for column 𝑗:

From (33),

so 𝑣 𝑗 moves in the direction that jointly benefits all queries that attend to it, weighted by their attention. In particular, if a small set of queries uses 𝑗 heavily, 𝑣 𝑗 becomes a prototype that serves that subset. This is the core specialization mechanism: values adapt to the error landscape created by their users.

Since 𝑔 𝑖 = 𝑘 𝛼 𝑖𝑘 𝑣 𝑘 , an infinitesimal change in 𝑣 𝑗 induces

Using 𝑢 𝑖 = 𝜕𝐿/𝜕𝑔 𝑖 , the first-order change in loss from this perturbation is

Interpretation. The factor 𝛼 𝑖 𝑗 says: the more query 𝑖 relies on value 𝑗, the more 𝑣 𝑗 ’s update affects 𝑔 𝑖 . The column weights {𝛼 𝑟 𝑗 } aggregate contributions from all queries that share 𝑗. The inner products 𝑢 ⊤ 𝑖 𝑢 𝑟 measure whether their error directions are aligned (helpful) or opposed (conflicting). If many 𝑢 𝑟 align with 𝑢 𝑖 , then 𝑣 𝑗 can move in a direction that helps them all. If some are anti-aligned, 𝑣 𝑗 must compromise, and specialization may become harder. Dominant-query approximation. If query 𝑖 is the primary user of key 𝑗 (i.e., 𝛼 𝑖 𝑗 ≫ 𝛼 𝑟 𝑗 for 𝑟 ≠ 𝑖), then 𝑟 𝛼 𝑟 𝑗 𝑢 𝑟 ≈ 𝛼 𝑖 𝑗 𝑢 𝑖 and

Thus, under this approximation, the update decreases loss for query 𝑖 by an amount proportional to 𝛼 2 𝑖 𝑗 ∥𝑢 𝑖 ∥ 2 : the stronger the attention and the larger the error, the bigger the immediate improvement.

Equations ( 23) and (33) together form a feedback loop:

(1) If 𝑣 𝑗 is particularly helpful to 𝑖 (large negative 𝑏 𝑖 𝑗 -E 𝛼 𝑖 [𝑏], meaning 𝑣 𝑗 points away from the error direction 𝑢 𝑖 ), then 𝑠 𝑖 𝑗 increases and 𝛼 𝑖 𝑗 grows. (2) Larger 𝛼 𝑖 𝑗 gives 𝑢 𝑖 more weight in ū𝑗 , so 𝑣 𝑗 moves further away from 𝑢 𝑖 (since Δ𝑣 𝑗 = -𝜂 ū𝑗 ).

(3) As 𝑣 𝑗 moves away from 𝑢 𝑖 , 𝑏 𝑖 𝑗 = 𝑢 ⊤ 𝑖 𝑣 𝑗 decreases further, increasing 𝐴 𝑖 𝑗 and reinforcing step 1. Repeated application yields specialized value vectors that serve distinct subsets of queries, while attention concentrates on these specialized prototypes.

The coupled dynamics derived above admit a useful analogy to Expectation-Maximization (EM), not as a literal optimization of an explicit latent-variable likelihood, but as a mechanistic correspondence between gradient flows and responsibility-weighted updates. Attention weights behave like soft responsibilities over latent sources, while value vectors act as prototypes updated under those responsibilities. Unlike classical EM, the updates here are driven by upstream gradients rather than observed data, and no standalone likelihood over values is optimized. We emphasize that this is an interpretive framework: we do not derive a surrogate objective with guaranteed monotonic improvement. The value lies in the geometric intuition it provides for understanding specialization dynamics.

For a fixed query 𝑖, the attention weights can be viewed as the posterior responsibilities of a latent variable 𝑍 𝑖 indicating which source position 𝑗 is active:

The score gradient

increases responsibilities for positions whose compatibility 𝑏 𝑖 𝑗 exceeds the current mean, just as the E-step in EM increases responsibilities for mixture components that explain the data well.

Given responsibilities 𝛼 𝑖 𝑗 , the value update

aggregates feedback from all assigned queries, weighted by their responsibilities. This is directly analogous to the M-step update for cluster means in Gaussian mixture models:

with 𝑢 𝑖 playing the role of residuals that need to be explained. A key difference from classical mixture-model EM is the role played by 𝑢 𝑖 . In standard EM, centroids are updated using observed data points. Here, 𝑢 𝑖 represents the residual error signal backpropagated from the loss. Values therefore move to better explain the error geometry induced by the current routing structure, rather than to maximize a likelihood over inputs. The analogy is structural rather than variational.

In classical EM, the E-step and M-step are separated: responsibilities are recomputed holding parameters fixed, then parameters are updated holding responsibilities fixed. In transformers trained with SGD, these steps are interleaved and noisy, but the first-order picture still resembles EM:

• E-step (routing): Score gradients adjust 𝑞 𝑖 and 𝑘 𝑗 so that attention patterns allocate responsibility to helpful positions. In practice, attention often stabilizes relatively early. • M-step (values): Value vectors continue to move under residual error signals, refining the content manifold even after attention appears frozen. This continues until calibration and likelihood converge.

Both updates occur simultaneously from each gradient computation. The “two-timescale” behavior emerges from relative gradient magnitudes and coupling structure, not from explicit alternation. We can make this connection more explicit by considering a “manual EM” algorithm that alternates:

(1) E-like step: Forward pass to compute 𝛼 𝑖 𝑗 using current 𝑞 𝑖 , 𝑘 𝑗 .

(2) M-like step on values: Update 𝑣 𝑗 with a closed-form gradient step using Δ𝑣 𝑗 = -𝜂 𝑖 𝛼 𝑖 𝑗 𝑢 𝑖 while freezing 𝑞 𝑖 , 𝑘 𝑗 . (3) Small SGD step on 𝑊 𝑄 ,𝑊 𝐾 ,𝑊 𝑂 : Update the nonlinear projection parameters with a small gradient step.

In Section 7.2, we compare such an EM-like schedule to standard SGD on a sticky Markov-chain task and find that both converge to similar solutions, but the EM-like updates reach low loss and sharp, focused attention much faster. Our analysis therefore lives at the EM/SGD level. However, our companion work [1] shows that the point-estimate parameters learned in this way support Bayesian computation in representation space: value manifolds, key frames, and query trajectories implement Bayesian belief updates in context. The present paper explains why cross-entropy and gradient descent naturally create these structures.

We now connect the gradient dynamics derived above to the geometric structures observed in Bayesian wind tunnels and production models.

In wind-tunnel experiments [1], which directly measure the geometric predictions made here:

• Early in training, attention entropy decreases and attention focuses on relevant hypotheses.

• Later in training, attention patterns appear stable, but value representations unfurl along a smooth curve; PC1 explains 84-90% of variance, strongly correlated with posterior entropy (|𝑟 | > 0.9). • Calibration error continues to drop even as attention maps remain visually unchanged.

Once score gradients have largely equalized compatibility (𝑏 𝑖 𝑗 ≈ E 𝛼 𝑖 [𝑏]), 𝜕𝐿/𝜕𝑠 𝑖 𝑗 ≈ 0 and attention freezes. But unless loss is exactly zero, 𝑢 𝑖 remains non-zero and value updates

continue, gradually aligning 𝑣 𝑗 along the principal directions of the residual error landscape. Under repeated updates, values come to lie on low-dimensional manifolds parameterized by downstream functionals such as posterior entropy.

The gradients to 𝑘 𝑗 show that keys are shaped by advantage signals:

If different subsets of queries consistently find different keys helpful, the corresponding gradient contributions push keys apart in 𝑘-space, encouraging approximate orthogonality of distinct hypothesis axes. Our wind-tunnel paper measures exactly this orthogonality: mean off-diagonal cosine similarity drops to 0.05-0.06 versus 0.08 for random vectors, a statistically significant reduction across multiple seeds (𝑝 < 0.001).

The empirically observed frame-precision dissociation-attention stably defining a hypothesis frame while calibration continues to improve-is now easy to interpret:

• The frame is defined by 𝑞 𝑖 , 𝑘 𝑗 and attention weights 𝛼 𝑖 𝑗 . This stabilizes when advantage signals equalize and 𝜕𝐿/𝜕𝑠 𝑖 𝑗 ≈ 0. • Precision lives in the arrangement of values 𝑣 𝑗 along manifolds that support fine-grained likelihood modeling. This continues to refine as long as 𝑢 𝑖 is non-zero.

Thus, a late-stage transformer has a fixed Bayesian frame (hypothesis axes and routing) but continues to sharpen its posterior geometry.

We now illustrate the theory with controlled simulations. All experiments use a single-head, single-layer attention block without residual connections or LayerNorm to keep the dynamics transparent.

We first consider a small toy setup with 𝑇 = 5, 𝑑 𝑥 = 3, 𝑑 𝑘 = 2, 𝑑 𝑣 = 2, and 𝐶 = 3 classes.

Setup.

• Projection matrices 𝑊 𝑄 ,𝑊 𝐾 ,𝑊 𝑉 ,𝑊 𝑂 are initialized with small Gaussian entries.

• Inputs 𝑥 𝑗 are drawn from N (0, 𝐼 ).

• Targets 𝑦 𝑖 are random in {0, 1, 2}.

• We run manual update steps using the closed-form gradients from Section 3, with a fixed learning rate.

Observations. Over ∼ 100 steps we observe:

(1) Attention heatmaps sharpen: mass concentrates on a few positions per query (Figure 2, Figure 3). ( 2) Value vectors move coherently in a low-dimensional subspace; their trajectories in a PCA projection show emergent manifold structure (Figure 5). (3) Cross-entropy loss decays smoothly (Figure 4), with most gains occurring as specialization emerges.

We next study a more structured task where attention can exploit temporal persistence: a sticky Markov chain over symbols.

Task. We generate sequences of length 𝑇 = 2000 over an 8-symbol vocabulary {0, . . . , 7} from a first-order Markov chain with self-transition probability 𝑃 (𝑦 𝑡 +1 = 𝑦 𝑡 | 𝑦 𝑡 ) = 0.3, otherwise transition to a different state with probability proportional to modulo distance of 7 with probability proportional to the distance from the previous state. With this circular distance, the nearby symbol gets higher probability than the distant one. Each symbol 𝑦 𝑡 has an associated mean embedding in R 20 ; the input at time 𝑡 is 𝑥 𝑡 = 𝜇 𝑦 𝑡 -1 + 𝜖 𝑡 with 𝜖 𝑡 ∼ N (0, 𝐼 ), so embeddings carry information about Protocols. We compare two training schemes with causal masking and matched total parameter updates per step:

(1) Standard SGD. Vanilla gradient descent on all parameters with learning rate 𝜂 = 0.01.

(2) EM-like schedule. Parameter-specific learning rates: 𝜂 𝑉 = 0.1 for value parameters, 𝜂 routing = 0.01 for 𝑊 𝑄 ,𝑊 𝐾 ,𝑊 𝑂 . All updates are computed from a single forward-backward pass and applied simultaneously. The EM-like schedule uses a larger learning rate for values (10×) to accelerate value specialization, compensating for the fact that value gradients are typically smaller in magnitude than routing gradients early in training. Both methods perform one gradient computation per step, with updates applied simultaneously (not alternating) using the respective learning rates.

• final cross-entropy loss, , Vol. 1, No. 1, Article . Publication date: January . • final accuracy,

• predictive entropy,

• KL(EM ∥ SGD) of the predictive distributions.

Results. Table 1 summarizes results across 5 random seeds after 1000 training steps. Figure 6-Figure 8 plot loss, accuracy, and entropy as a function of training step. EM-like training converges significantly faster than SGD: it reaches SGD’s final loss in under half the steps (2.3× speedup), with loss and entropy dropping sharply while accuracy rises quickly. This acceleration occurs because the EM-motivated learning rate schedule (with larger LR for values) allows values to specialize more rapidly, whereas uniform SGD must balance routing and content updates at the same rate. Even with this single-layer, single-head architecture, both methods approach the theoretical Bayesian minimum (1.829 bits), with EM converging closer (1.998 vs 2.077 bits entropy). The residual gap of ∼0.17 bits from the theoretical minimum likely reflects the limited capacity of a single-head architecture to fully capture the transition structure; concurrent work [1] shows that deeper architectures close this gap to < 0.01 bits on similar tasks. To compare micro behavior of EM and Takeaways. Both EM-like and SGD training ultimately converge toward similar qualitative solutions: specialized values and focused attention. However, the EM-like schedule reaches this state with fewer steps and sharper specialization. This matches the two-timescale story: responsibilities (attention) can be treated as approximately converged, and closed-form value updates can exploit this stability to accelerate manifold formation.

The gradient analysis suggests useful diagnostics and design principles for training and interpreting transformer attention.

• Compatibility matrix 𝐵 = (𝑏 𝑖 𝑗 ). Monitoring 𝑏 𝑖 𝑗 = 𝑢 ⊤ 𝑖 𝑣 𝑗 reveals which values are most helpful to which queries.

). The sign and magnitude predict where attention will strengthen or weaken. • Column usage 𝑖 𝛼 𝑖 𝑗 . Low column sums identify underused or dead values.

• Value norms ∥𝑣 𝑗 ∥. Norm trajectories can flag potential instability (exploding or vanishing norms).

• LayerNorm on values can stabilize norms while leaving directional dynamics intact.

• Attention dropout disrupts the feedback loop, limiting over-specialization and encouraging more evenly used values. • Learning rate choices modulate the timescale separation between routing and content; smaller learning rates make the first-order picture more accurate.

• Multi-head attention allows multiple specialized routing manifolds, reducing competition within a single head. • Depth naturally supports the binding-elimination-refinement hierarchy observed in our wind-tunnel experiments and large models. • Residual connections help maintain useful intermediate representations even as individual heads specialize strongly.

The gradient dynamics derived in Sections 4 and 5 are specific to softmax attention. However, companion work shows that selective state-space models (Mamba) achieve comparable Bayesian inference through a different routing mechanism-input-dependent gating rather than query-key matching. This raises a natural question: is there a unifying principle that explains why both architectures develop Bayesian geometry? We conjecture that the essential structure is contentbased value routing, and that any mechanism satisfying this property will exhibit similar gradient dynamics under cross-entropy training. The key requirement is that routing weights depend on what is at each position, not just where it is. This excludes:

• Fixed positional patterns (e.g., “always attend to position 𝑡 -1”) • Learned but content-independent weights (e.g., sinusoidal position encodings alone) • Architectures where gating is independent of cross-position relationships (e.g., standard LSTM gates)

Softmax Attention. The standard transformer attention mechanism is a content-based routing mechanism with:

𝑣 𝑗 = 𝑊 𝑉 𝑥 𝑗 (46)

Routing weights depend on content through the query-key inner product 𝑞 ⊤ 𝑖 𝑘 𝑗 .

If Conjectures 9.2 and 9.3 hold generally, they explain why both attention and selective SSMs develop Bayesian geometry:

(1) Routing specialization: Advantage-based updates cause routing to specialize-weights increase for value positions that are “above average” at reducing the loss for each query. This explains the empirical finding from Paper 1: transformers and Mamba both achieve Bayesian inference (0.049 and 0.031 bits MAE on HMM tracking), while LSTMs fail (0.416 bits). The LSTM lacks content-based routing and therefore cannot develop the coupled specialization that produces Bayesian geometry.

A complete theory requires deriving the gradient dynamics for selective SSMs explicitly. The key challenges are:

(1) Sequential dependencies: Unlike attention, SSM gradients must backpropagate through the recurrence ℎ 𝑡 = 𝑓 (ℎ 𝑡 -1 , 𝑥 𝑡 , Δ 𝑡 , 𝐵 𝑡 ). (2) Coupled parameters: The input-dependent Δ, 𝐵, 𝐶 are computed through shared projections, creating coupled dynamics. (3) Continuous vs. discrete: SSMs operate in continuous state-space; the EM analogy may require a continuous relaxation (e.g., variational inference interpretation).

We leave formal derivation for future work, but note that the empirical success of Mamba on Bayesian tasks strongly suggests that similar dynamics emerge. The abstract framework proposed here provides a roadmap for unifying the theory of content-based routing across architectures.

10 Related Work

Several works argue that transformers implement approximate Bayesian inference, either behaviorally or via probing [e.g. 6, 7]. Our companion paper [1] demonstrates exact Bayesian behavior in small wind tunnels, showing that architectures with content-based value routing (transformers and Mamba) succeed while those without it (LSTMs, MLPs) fail. A scaling paper shows similar geometric signatures in production LLMs. The present work explains how gradient dynamics in attention produce these geometries-deriving analogous dynamics for selective state-space models remains future work.

Mechanistic interpretability studies identify specific heads and circuits performing copy, induction, and other algorithmic tasks [3,4]. Our framework complements this line by explaining how specialization arises from the interaction between routing and content, rather than treating specialized heads as primitive.

The implicit bias of gradient descent in linear and deep networks has been extensively studied [2,5]. We extend these ideas to attention: gradient descent implicitly favors representations where routing aligns with error geometry and values lie on low-dimensional manifolds that support Bayesian updating.

Attention Training Dynamics. Recent work has analyzed attention optimization from complementary perspectives. Max-margin analyses show directional convergence of attention parameters toward SVM-like solutions. Two-stage dynamics have been identified where early training exhibits parameter condensation followed by key/query activation. Our advantage-based routing perspective complements these findings: the baseline-subtracted compatibility gradient we derive is the mechanism by which attention reallocates mass, and our two-timescale observation (fast routing, slow values) aligns with reports of stage-wise dynamics. The key difference is our focus on the EMlike interpretation and the connection to Bayesian manifold formation. The responsibility-weighted value updates derived here are reminiscent of neural EM and slot-attention models, where soft assignments drive prototype updates. The key distinction is that, in transformers, responsibilities are computed via content-addressable attention and prototype updates are driven by backpropagated error signals rather than reconstruction likelihoods. Our focus is not on proposing a new EM-style architecture, but on showing how standard cross-entropy training in attention layers induces EM-like specialization dynamics as a consequence of gradient flow.

Our analysis is deliberately minimal and controlled.

First-order approximation. We work in a first-order regime, assuming small learning rates and ignoring higher-order and stochastic effects (e.g., momentum, Adam, mini-batch noise). Extending the analysis to realistic optimizers is an important next step.

Single-head, single-layer focus. We analyze a single head in isolation, without residual pathways or LayerNorm. Multi-head, multi-layer dynamics-including inter-head coordination and hierarchical specialization-remain open.

Finite vs. infinite width. We do not explicitly connect our analysis to the neural tangent kernel or infinite-width limits. Bridging these regimes may help clarify when transformers operate in a feature-learning versus lazy-training mode.

Large-scale empirical validation. Our toy simulations are intentionally small. Applying the diagnostics in Section 8 to full-scale LLM training runs, tracking advantage matrices and manifold formation over time, is a promising direction.

Attention-specific analysis. Our gradient derivations are specific to softmax attention. Companion work shows that selective state-space models (Mamba) also implement Bayesian inference via content-based routing, but through different mechanics-input-dependent gating rather than querykey matching. Deriving analogous gradient dynamics for selective SSMs, and identifying whether a similar EM-like structure emerges, is an important direction for unifying the theory of content-based routing across architectures.

Paper I establishes what architectures can implement Bayesian inference, taxonomized by three inference primitives: belief accumulation, belief transport, and random-access binding. This paper explains how gradient descent learns to implement these primitives. Our key findings are:

, Vol. 1, No. 1, Article . Publication date: January .