Hierarchical Latent Structures in Data Generation Process Unify Mechanistic Phenomena across Scale

Contemporary studies have uncovered many puzzling phenomena in the neural information processing of Transformer-based language models. Building a robust, unified understanding of these phenomena requires disassembling a model within the scope of its training. While the intractable scale of pretraining corpora limits a bottom-up investigation in this direction, simplistic assumptions of the data generation process limit the expressivity and fail to explain complex patterns. In this work, we use probabilistic context-free grammars (PCFGs) to generate synthetic corpora that are faithful and computationally efficient proxies for web-scale text corpora. We investigate the emergence of three mechanistic phenomena: induction heads, function vectors, and the Hydra effect, under our designed data generation process, as well as in the checkpoints of real-world language models. Our findings suggest that hierarchical structures in the data generation process serve as the X-factor in explaining the emergence of these phenomena. We provide the theoretical underpinnings of the role played by hierarchy in the training dynamics of language models. In a nutshell, our work is the first of its kind to provide a unified explanation behind the emergence of seemingly unrelated mechanistic phenomena in LLMs, augmented with efficient synthetic tooling for future interpretability research.

💡 Research Summary

This paper investigates why several seemingly unrelated mechanistic phenomena—induction heads, function vectors, and the Hydra effect—appear simultaneously in Transformer‑based large language models (LLMs). The authors hypothesize that the hierarchical structure of the training data, rather than model architecture or optimization alone, is the key driver (“X‑factor”) behind their co‑emergence.

To test this, they construct two synthetic data generation pipelines that are matched on surface statistics (vocabulary size, token frequency distribution, sentence length) but differ in structural complexity. The first pipeline is a Zipf‑based N‑gram generator, which provides only flat, sequential dependencies. The second pipeline uses a probabilistic context‑free grammar (PCFG) that explicitly encodes hierarchical, recursive relations among non‑terminals, mimicking the subject‑verb‑object structure of natural language and allowing document‑level shuffling.

Identical Transformer models (same depth, width, optimizer, learning‑rate schedule) are trained on each corpus, and checkpoints are evaluated at regular intervals. Three phenomena are quantified:

1. k‑order induction heads – attention heads that copy a token (or a sequence of length k) when the exact context reappears later. The authors compute a generalized prefix‑score for k = 1…10. In the PCFG‑trained models, induction‑related attention sharply rises after ~6 k training steps for all k, whereas N‑gram models never develop such heads.

2. Function vectors – low‑dimensional representations that encode a task‑specific input‑output mapping. By extracting a vector from a few‑shot context and patching it into a zero‑shot query, they measure the increase in the correct logit. Function‑vector strength emerges concurrently with induction heads in the PCFG models (around 6 k steps) and is absent in N‑gram models.

3. Hydra effect – the phenomenon where ablating a layer (zero‑ing its output) leads to a compensatory increase in predictive power of a subsequent layer. The authors define Δ̄(ℓ)ᵐ as the average drop in ground‑truth logits at layer ℓ when layer ℓ‑m is removed. In PCFG models, a modest effect appears early, but after 6 k steps the compensation becomes strong, matching or exceeding that observed in a real‑world 1‑billion‑parameter model (OLMo‑1B). N‑gram models show no compensation.

To link these behavioral findings to internal representations, the authors train a structural probe (Hewitt & Manning, 2019) that learns a linear map B such that Euclidean distances between transformed token embeddings approximate tree‑distance in the ground‑truth parse. The unlabeled undirected attachment score (UUAS) rises dramatically for PCFG‑trained models, indicating that the model’s hidden states encode the hierarchical geometry of the data.

The paper also offers a theoretical sketch: hierarchical productions create gradients that concentrate learning signals on specific layers, simultaneously encouraging (i) local pattern‑copy mechanisms (induction heads), (ii) localized mapping mechanisms (function vectors), and (iii) global redundancy‑compensation mechanisms (Hydra effect). Recursive non‑terminals force the model to distinguish “pattern‑replication” from “pattern‑inference,” a capability that flat Markovian data cannot induce.

Key contributions:

• Demonstrates empirically that hierarchical data generation is sufficient to reproduce all three mechanistic phenomena, while a flat N‑gram baseline fails.
• Shows that a PCFG‑based synthetic corpus faithfully mirrors the learning dynamics of a real‑world LLM, providing a cost‑effective testbed for interpretability research.
• Provides a unified explanatory framework linking data hierarchy, training dynamics, and emergent mechanisms across different scales.

Overall, the work highlights the often‑overlooked role of data structure in shaping the internal circuitry of LLMs and suggests that future interpretability studies should consider hierarchical synthetic data as a primary investigative tool.