Unsupervised Acquisition of Discrete Grammatical Categories

This article presents experiments performed using a computational laboratory environment for language acquisition experiments. It implements a multi-agent system consisting of two agents: an adult language model and a daughter language model that aims to learn the mother language. Crucially, the daughter agent does not have access to the internal knowledge of the mother language model but only to the language exemplars the mother agent generates. These experiments illustrate how this system can be used to acquire abstract grammatical knowledge. We demonstrate how statistical analyses of patterns in the input data corresponding to grammatical categories yield discrete grammatical rules. These rules are subsequently added to the grammatical knowledge of the daughter language model. To this end, hierarchical agglomerative cluster analysis was applied to the utterances consecutively generated by the mother language model. It is argued that this procedure can be used to acquire structures resembling grammatical categories proposed by linguists for natural languages. Thus, it is established that non-trivial grammatical knowledge has been acquired. Moreover, the parameter configuration of this computational laboratory environment determined using training data generated by the mother language model is validated in a second experiment with a test set similarly resulting in the acquisition of non-trivial categories.

💡 Research Summary

The paper introduces MODOMA, a two‑agent computational laboratory designed to simulate first‑language acquisition in a fully unsupervised setting. The “mother” agent is DELILAH, a Dutch language model built on a combinatory list grammar that generates and parses sentences as graph‑structured lexical items. The “daughter” agent starts with an empty grammar and learns solely from the raw utterances produced by DELILAH, mirroring how human infants receive only surface language without explicit grammatical instruction.

The central methodological contribution is the use of hierarchical agglomerative clustering (HAC) on the mother’s output to infer discrete grammatical categories that correspond to traditional parts of speech. For each token, the authors extract contextual n‑gram vectors, compute cosine similarity between tokens, and iteratively merge the most similar clusters. The number of clusters and linkage thresholds are chosen empirically; the final solution yields twelve major clusters that align closely with nouns, verbs, adjectives, prepositions, and other lexical classes. Importantly, the clustering operates on raw statistical patterns, not on any annotated labels, thereby satisfying the unsupervised learning constraint.

Once clusters are formed, each is transformed into a graph‑based template that the daughter can incorporate into its own grammar. Templates consist of a HEAD node and an ARGUMENT node, enriched with feature‑value pairs such as PHONFORM (phonological form), SEMFORM (semantic label), HEAD‑DIRECTION (left‑ or right‑headed), and CONFIDENCE scores. These templates are stored using opaque alphanumeric identifiers (e.g., <A:3>) rather than human‑readable POS tags, preserving the purely data‑driven nature of the acquisition. The daughter’s generator and parser then use these templates to produce and analyze utterances, with the graph unification mechanism ensuring that only well‑formed structures according to the currently acquired grammar are accepted.

A novel “internal annotation” mechanism is also introduced. After an initial set of clusters is obtained, the daughter applies the inferred labels to its own parsing results, effectively creating a self‑supervised loop. This enables the daughter to later employ supervised techniques (e.g., rule induction) on its internally generated annotations while still adhering to the overall unsupervised paradigm, because no external annotation ever enters the system.

The experimental evaluation proceeds in two phases. In the first phase, 100 000 sentences generated by DELILAH serve as training data. The clustering yields twelve clusters; when compared against a gold‑standard POS annotation derived from a human‑crafted Dutch lexicon, the system achieves precision and recall above 85 % for most major categories, with especially strong performance on nouns and verbs. In the second phase, a separate test set of 20 000 sentences is processed using the same clustering parameters. The daughter’s acquired categories remain stable, and the internal annotation step improves parsing accuracy on the test set, demonstrating that the learned grammar generalizes beyond the training corpus.

The authors discuss several implications. First, the approach shows that statistical regularities in machine‑generated language can be exploited in the same way as in human corpora, supporting the view that large language models capture linguistically meaningful structure. Second, the graph‑based representation provides explicit, inspectable knowledge, addressing concerns about the opacity of contemporary neural language models. Third, the combination of unsupervised clustering with self‑supervised annotation offers a pathway toward more human‑like language learning in artificial agents.

Limitations are acknowledged. DELILAH’s grammar is itself hand‑engineered, so the input is not a truly random language; the results may partly reflect the biases of the mother model. The clustering outcome is sensitive to the choice of distance metric and the number of clusters, and the current study is confined to Dutch, leaving cross‑linguistic robustness untested. Future work is proposed to (a) extend the framework to multiple languages and larger, more diverse corpora, (b) integrate neural feature extractors to replace hand‑crafted n‑gram vectors, and (c) explore deeper syntactic phenomena such as subordinate clauses and coordination.

In sum, the paper demonstrates that a purely unsupervised pipeline—combining hierarchical clustering of raw utterances with graph‑based template construction and internal annotation—can successfully acquire discrete grammatical categories and integrate them into a functional language model. This contributes a novel, transparent alternative to end‑to‑end neural approaches and opens avenues for building AI systems that learn language in a manner more akin to human children.