Constructing BERT Models: How Team Dynamics and Focus Shape AI Model Impact

The rapid evolution of AI technologies, exemplified by BERT-family models, has transformed scientific research, yet little is known about their production and recognition dynamics in the scientific system. This study investigates the development and impact of BERT-family models, focusing on team size, topic specialization, and citation patterns behind the models. Using a dataset of 4,208 BERT-related papers from the Papers with Code (PWC) dataset, we analyze how the BERT-family models evolve across methodological generations and how the newness of models is correlated with their production and recognition. Our findings reveal that newer BERT models are developed by larger, more experienced, and institutionally diverse teams, reflecting the increasing complexity of AI research. Additionally, these models exhibit greater topical specialization, targeting niche applications, which aligns with broader trends in scientific specialization. However, newer models receive fewer citations, particularly over the long term, suggesting a “first-mover advantage,” where early models like BERT garner disproportionate recognition. These insights highlight the need for equitable evaluation frameworks that value both foundational and incremental innovations. This study underscores the evolving interplay between collaboration, specialization, and recognition in AI research.

💡 Research Summary

This paper investigates the production and recognition dynamics of the BERT family of language models, focusing on how team composition, topical specialization, and citation impact evolve across methodological generations. Using a combined dataset from Papers with Code (PWC) and OpenAlex, the authors identified 4,208 papers that genuinely discuss BERT‑related models. To ensure relevance, they fine‑tuned a BERT‑base‑uncased classifier on PWC abstracts and validated it against a human‑labeled set, achieving 0.97 accuracy and 0.95 F1‑score.

Methodologically, the study introduces a “generation score” that captures a model’s position in the knowledge flow network rather than simply its publication year. Citation links among BERT‑related concepts (e.g., BERT → BioBERT) form a directed graph; a graph‑hierarchy algorithm assigns each concept a generation level, and a paper’s generation score is the average of the concepts appearing in its abstract. This metric allows the authors to disentangle chronological effects from intellectual dependency effects.

Three research questions guide the analysis: (1) Do newer BERT models require larger, more diverse teams? (2) Are newer models more topically narrow? (3) Do newer models receive comparable citation recognition? Corresponding hypotheses (including sub‑hypotheses on author count and institutional diversity) are tested using multivariate regressions. Dependent variables are grouped into (a) research inputs (author count, average career age, prior citations, institution count, institutional productivity, Herfindahl‑based dispersion, country count, international dispersion), (b) conceptual breadth (number of OpenAlex concepts attached to a paper), and (c) impact (citations accrued in the first, second, and third year after publication). Control variables include venue type (conference vs. journal), core‑source status, impact factor, industrial affiliation, authors’ prior BERT publications, field fixed effects (19 Level‑0 domains), and year fixed effects. Count outcomes are modeled with negative binomial regression; continuous outcomes (e.g., average career age) use OLS.

Key findings support all three hypotheses. Papers introducing newer BERT variants involve significantly larger teams: author counts rise by roughly 30 % on average, and the number of distinct institutions shows a comparable increase. The average career age of contributors is 2–3 years higher, and authors bring more cumulative citations, indicating greater experience and expertise. In terms of topical scope, newer models are associated with about 15 % fewer OpenAlex concepts, reflecting a trend toward domain‑specific specialization (e.g., BioBERT, FinBERT). Citation analysis reveals a consistent disadvantage for newer models: one‑year citations are about 20 % lower, two‑year citations about 30 % lower, and three‑year citations roughly 35 % lower than those of the original BERT. This pattern confirms a “first‑mover advantage” where foundational models retain disproportionate scholarly attention over time. Additional observations note that papers with industrial affiliations tend to receive fewer citations, while publications in core conferences or high‑impact journals garner more citations, aligning with established bibliometric patterns.

The authors conclude that AI research is becoming increasingly collaborative, resource‑intensive, and specialized, yet the scholarly reward system remains skewed toward early, groundbreaking models. They argue for more equitable evaluation frameworks that recognize both foundational contributions and incremental, domain‑specific innovations. Limitations include the focus on BERT‑related literature (potentially missing broader AI trends) and reliance on citation counts as the sole impact metric; future work should incorporate alternative impact indicators such as code reuse, industry adoption, and social media attention, as well as extend the analysis to other model families (e.g., GPT, T5) for comparative insight.