A large-scale pipeline for automatic corpus annotation using LLMs: variation and change in the English consider construction

As natural language corpora expand at an unprecedented rate, manual annotation remains a significant methodological bottleneck in corpus linguistic work. We address this challenge by presenting a scalable pipeline for automating grammatical annotation in voluminous corpora using large language models (LLMs). Unlike previous supervised and iterative approaches, our method employs a four-phase workflow: prompt engineering, pre-hoc evaluation, automated batch processing, and post-hoc validation. We demonstrate the pipeline’s accessibility and effectiveness through a diachronic case study of variation in the English evaluative consider construction (consider X as/to be/zero Y). We annotate 143,933 ‘consider’ concordance lines from the Corpus of Historical American English (COHA) via the OpenAI API in under 60 hours, achieving 98 percent+ accuracy on two sophisticated annotation procedures. A Bayesian multinomial GAM fitted to 44,527 true positives of the evaluative construction reveals previously undocumented genre-specific trajectories of change, enabling us to advance new hypotheses about the relationship between register formality and competing pressures of morphosyntactic reduction and enhancement. Our results suggest that LLMs can perform a range of data preparation tasks at scale with minimal human intervention, unlocking substantive research questions previously beyond practical reach, though implementation requires attention to costs, licensing, and other ethical considerations.

💡 Research Summary

The paper tackles the growing methodological bottleneck in corpus linguistics: the need to manually annotate massive amounts of textual data. While modern corpora such as COHA, NOW, and the TenTen family contain billions of words, extracting linguistically relevant subsets still requires human expertise, especially for constructions that involve subtle semantic and syntactic judgments. The authors propose a scalable, four‑phase pipeline that leverages large language models (LLMs) to automate grammatical annotation with minimal human supervision.

Phase 1, Prompt Engineering, encodes the classification criteria for the English evaluative “consider” construction, including detailed definitions, illustrative examples, edge cases, and a strict output format. Phase 2, Pre‑hoc Evaluation, tests the prompt on multiple independent random samples to confirm its reliability before large‑scale deployment. Phase 3, Automated Batch Processing, uses the OpenAI GPT‑5 API to classify 143,933 concordance lines containing forms of consider from the Corpus of Historical American English (COHA). Each line undergoes two sequential queries: first to decide whether the use is evaluative or non‑evaluative, and second to assign the complement type (zero, to‑be, or as). The entire batch is processed in under 60 hours, at roughly 9 seconds per line, and costs about US $104 in API fees. Phase 4, Post‑hoc Validation, employs stratified random sampling across COHA’s twenty decades and five genres (fiction, newspaper, academic, spoken, and advertising) to verify that overall accuracy exceeds 98 % for both classification tasks.

The pipeline reduces the dataset from 99,406 tokens to 44,527 evaluative instances, demonstrating that LLMs can efficiently isolate minority phenomena that would otherwise require exhaustive manual inspection. Accuracy analysis shows that most errors stem from ambiguous contexts, passive constructions, or rare contemporary usages that the model misinterprets.

With the cleaned set of evaluative instances, the authors fit a Bayesian multinomial Generalized Additive Model (GAM) to examine diachronic change in complement type distribution. The model uncovers previously undocumented, genre‑specific trajectories: formal registers (e.g., academic prose, newspapers) show a steady decline of the to‑be complement and a rise of the zero complement, whereas informal registers (spoken language, advertising) exhibit a temporary increase in the as complement during the mid‑20th century. These patterns support a new hypothesis that register formality interacts with competing pressures of morphosyntactic reduction (favoring zero complement) and enhancement (favoring overt markers like as).

Beyond the substantive linguistic findings, the study highlights several methodological implications. First, the pipeline requires only basic scripting to call the API; the intellectual heavy lifting is front‑loaded into prompt design and validation, making sophisticated annotation accessible to scholars without deep programming expertise. Second, the cost‑effectiveness (≈ $0.0007 per annotation) demonstrates that large‑scale LLM‑assisted annotation is financially viable even for modestly funded projects. Third, the authors discuss ethical and practical considerations, including model version drift, licensing constraints on the underlying corpora, and the need for transparent reporting of API usage and error rates.

In the discussion, the authors compare their approach to traditional supervised machine‑learning pipelines, noting that LLMs eliminate the need for large labeled training sets while still achieving comparable or superior accuracy. They also outline future directions: integrating multiple LLMs in an ensemble to further boost reliability, automating prompt optimization through meta‑learning, and extending the framework to other languages and construction types.

In conclusion, the paper provides a robust, reproducible workflow that demonstrates how contemporary LLMs can overcome the annotation bottleneck in large corpora, enabling researchers to ask substantive diachronic and variationist questions that were previously out of reach. The successful case study of the English evaluative consider construction illustrates both the technical feasibility and the theoretical payoff of this AI‑augmented methodology.