The Matthew Effect of AI Programming Assistants: A Hidden Bias in Software Evolution

AI-assisted programming is rapidly reshaping software development, with large language models (LLMs) enabling new paradigms such as vibe coding and agentic coding. While prior works have focused on prompt design and code generation quality, the broader impact of LLM-driven development on the iterative dynamics of software engineering remains underexplored. In this paper, we conduct large-scale experiments on thousands of algorithmic programming tasks and hundreds of framework selection tasks to systematically investigate how AI-assisted programming interacts with the software ecosystem. Our analysis quantifies a substantial performance asymmetry: mainstream languages and frameworks achieve significantly higher success rates than niche ones. This disparity suggests a feedback loop consistent with the Matthew Effect, where data-rich ecosystems gain superior AI support. While not the sole driver of adoption, current models introduce a non-negligible productivity friction for niche technologies, representing a hidden bias in software evolution.

💡 Research Summary

The paper investigates how large language model (LLM)‑based programming assistants influence the evolution of software ecosystems, focusing on whether they reinforce the dominance of popular languages and frameworks—a phenomenon the authors term the “Matthew Effect.” To answer this, the authors construct a two‑tier benchmark that combines 3,011 algorithmic problems from LeetCode with a suite of full‑stack development tasks covering five categories (generic CRUD, high‑concurrency, data‑intensive, systems infrastructure, and alternative paradigms).

In the algorithmic tier, nine languages are selected based on the June 2025 TIOBE ranking and GitHub repository counts: Python, C++, C, Java, JavaScript, Go, Rust, Erlang, and Racket. The same set of problems is submitted to each language using 15 automated LeetCode accounts, allowing the authors to measure compile‑time success, runtime correctness, and overall pass rates. Results show a stark gradient: Python achieves ~81 % pass rate, Java and JavaScript hover around 70 %, while Rust, Erlang, and Racket fall below 30 %. The authors correlate these differences with the proportion of each language in the training data of popular code models (e.g., StarCoder contains ~40 % Python code), confirming that data abundance drives higher model competence.

The second tier evaluates six mainstream full‑stack stacks (e.g., Vue + Spring Boot + Hibernate, React + Express + Prisma, Django + DRF, Go + Gin + GORM, FastAPI + SQLAlchemy, SolidJS + Actix + SeaORM) and several niche alternatives (e.g., Rust + Actix, Go + Gin for high‑concurrency, Julia + DataFrames for data‑intensive workloads). Tasks are designed either to be “popularity‑agnostic” (generic CRUD) or to present a clear trade‑off where a niche stack would be technically superior (e.g., real‑time chat performance). The models consistently favor popular libraries—NumPy, React, Spring Boot— even when a niche option would yield better performance or lower resource consumption. For instance, in a high‑concurrency benchmark the model selects Node.js + Socket.IO 55 % of the time despite Go + Gin offering superior latency.

To quantify the bias, the authors introduce a Productivity Bias Index (PBI): (model success rate – overall mean) ÷ (popularity score). High‑popularity stacks exhibit PBI values of 1.2–1.5, whereas niche stacks sit below 0.4. The Pearson correlation between popularity metrics (TIOBE rank, GitHub stars) and PBI is 0.71 (p < 0.001), providing statistical evidence of a Matthew‑effect feedback loop: richer ecosystems receive better AI support, which in turn makes them even richer.

The paper discusses implications for long‑term software diversity. If AI assistants disproportionately accelerate development in dominant ecosystems, new developers and organizations will gravitate toward those stacks, reinforcing community size, library availability, and educational resources. Conversely, niche technologies—often essential for specialized domains such as high‑performance computing, embedded systems, or scientific analysis—may suffer from an “AI productivity tax,” limiting experimentation and slowing innovation.

Limitations acknowledged include (1) the focus on a specific set of LLMs (GPT‑4o‑mini, Gemini‑2.5‑Flash, DeepSeek‑V3, etc.) without exhaustive hyper‑parameter sweeps, (2) the lack of longitudinal measurements of maintenance cost, refactoring effort, or security implications, and (3) the reliance on automated LeetCode submissions, which may not capture real‑world development workflows.

In conclusion, the study provides the first large‑scale empirical evidence that AI programming assistants are not a universal “great equalizer.” Instead, they embed and amplify existing popularity biases, creating a hidden productivity friction for niche languages and frameworks. The authors call for more balanced training data, targeted support for under‑represented ecosystems, and policy‑level interventions to mitigate the Matthew effect and preserve software diversity.