Deep networks learn to parse uniform-depth context-free languages from local statistics

Understanding how the structure of language can be learned from sentences alone is a central question in both cognitive science and machine learning. Studies of the internal representations of Large Language Models (LLMs) support their ability to parse text when predicting the next word, while representing semantic notions independently of surface form. Yet, which data statistics make these feats possible, and how much data is required, remain largely unknown. Probabilistic context-free grammars (PCFGs) provide a tractable testbed for studying these questions. However, prior work has focused either on the post-hoc characterization of the parsing-like algorithms used by trained networks; or on the learnability of PCFGs with fixed syntax, where parsing is unnecessary. Here, we (i) introduce a tunable class of PCFGs in which both the degree of ambiguity and the correlation structure across scales can be controlled; (ii) provide a learning mechanism – an inference algorithm inspired by the structure of deep convolutional networks – that links learnability and sample complexity to specific language statistics; and (iii) validate our predictions empirically across deep convolutional and transformer-based architectures. Overall, we propose a unifying framework where correlations at different scales lift local ambiguities, enabling the emergence of hierarchical representations of the data.

💡 Research Summary

The paper tackles a fundamental question in both cognitive science and machine learning: how can a system infer the hierarchical structure of a language from raw sentences alone? To answer this, the authors use probabilistic context‑free grammars (PCFGs) as a tractable testbed, but they move beyond previous work that either examined post‑hoc parsing‑like behavior of trained networks or studied PCFGs with a fixed syntax where parsing is unnecessary.

Key contributions

1. A controllable family of PCFGs – the “varying‑tree Random Hierarchy Model” (RHM). Each grammar has a uniform depth L, a fixed vocabulary size v at every non‑terminal level, and a random set of binary (m₂) and ternary (m₃) production rules. By scaling m₂ and m₃ with v (m₂ = f₂·v, m₃ = f₃·v²) and by tuning the rule density f, the authors can precisely control the amount of global ambiguity: the probability that a given sentence can be generated with more than one root label. When f is small the grammar is almost unambiguous; when f exceeds a critical value f_c ≈ 3/8 the grammar becomes highly ambiguous, making root classification impossible.

2. A learning mechanism inspired by deep convolutional networks – the authors show that the classic inside algorithm (or its Boolean version, CYK) can be expressed as a set of tensors M^{(ℓ)}_{i,λ}(z) that record, for each span (i, λ), which non‑terminal z can generate that substring. These tensors are built recursively from the bottom up, exactly mirroring the hierarchical receptive fields of CNNs and the attention‑driven composition in Transformers. The paper argues that during supervised training on the root‑label classification task, a deep network implicitly learns to approximate these tensors by clustering tokens that exhibit strong statistical correlations across scales.

3. Theoretical sample‑complexity analysis – In the low‑ambiguity regime (f < f_c) the task reduces to correctly predicting the root label α given a sentence x. The authors derive a scaling law for the minimal number of training sentences P* required to reliably detect the correlation between α and the local substrings that the network aggregates. The result is a power‑law dependence:
   P* = Θ( v·m₃^{β(L)} ),
   where β(L) grows with the depth L and captures how many hierarchical levels must be resolved. Intuitively, each rule must be observed enough times for the network to estimate its contribution to the root label.

Empirical validation
The authors generate synthetic corpora from the varying‑tree RHM for a range of vocabulary sizes (v = 8, 10, 15, 20) and depths (L = 2 … 6). They train two architectures: a deep 1‑D convolutional network and a standard Transformer encoder. The training objective is a multi‑class cross‑entropy loss over the v possible root labels.

Results show two striking phenomena:

• Scaling collapse – When the number of training sentences P is plotted against the loss, curves for all (v, L) settings collapse onto a single master curve after rescaling P by the theoretical P*. This confirms that the derived sample‑complexity law accurately predicts how much data each architecture needs, regardless of model depth or vocabulary size.

• Ambiguity transition – Varying the rule density f reveals a sharp increase in loss around f ≈ 0.35–0.40, precisely the predicted critical point f_c = 3/8. Above this threshold, the network’s accuracy drops to chance level (1/v), indicating that the global ambiguity makes the root label unrecoverable.

Both CNNs and Transformers exhibit similar data efficiency, though Transformers achieve slightly higher final accuracies, suggesting that the underlying “inside‑like” computation is architecture‑agnostic.

Implications
The work provides a concrete bridge between statistical properties of the training data (rule density, vocabulary size, depth) and the learning dynamics of deep networks. It explains why large language models can acquire syntactic knowledge from relatively modest corpora: natural languages occupy the low‑ambiguity region (small effective f) and contain rich multi‑scale correlations that deep networks can exploit. Moreover, the analysis shows that the internal mechanisms of CNNs and Transformers can be interpreted as approximations of classic parsing algorithms, offering a principled explanation for the emergence of hierarchical representations observed in recent interpretability studies.

Future directions include extending the model to non‑uniform depths, incorporating semantic labels, and testing the theory on real‑world corpora to see whether natural language indeed lies below the predicted ambiguity threshold. Overall, the paper delivers a unified theoretical‑empirical framework that clarifies how local statistical regularities can give rise to global hierarchical parsing abilities in deep neural networks.