Is there "Secret Sauce'' in Large Language Model Development?

Do leading LLM developers possess a proprietary ``secret sauce’’, or is LLM performance driven by scaling up compute? Using training and benchmark data for 809 models released between 2022 and 2025, we estimate scaling-law regressions with release-date and developer fixed effects. We find clear evidence of developer-specific efficiency advantages, but their importance depends on where models lie in the performance distribution. At the frontier, 80-90% of performance differences are explained by higher training compute, implying that scale–not proprietary technology–drives frontier advances. Away from the frontier, however, proprietary techniques and shared algorithmic progress substantially reduce the compute required to reach fixed capability thresholds. Some companies can systematically produce smaller models more efficiently. Strikingly, we also find substantial variation of model efficiency within companies; a firm can train two models with more than 40x compute efficiency difference. We also discuss the implications for AI leadership and capability diffusion.

💡 Research Summary

The paper investigates whether leading developers of large language models (LLMs) possess a proprietary “secret sauce” that gives them a systematic advantage beyond raw compute scaling. Using a dataset of 809 LLMs released between October 2022 and March 2025, the authors collect MMLU‑Pro benchmark scores and training compute (derived from parameters and token counts). They estimate a log‑log regression of logit‑transformed performance on log compute, adding three sets of fixed effects: (i) publication‑period dummies to capture shared algorithmic/technical progress, (ii) developer dummies to capture company‑specific efficiency (the “secret sauce”), and (iii) a residual term that absorbs model‑specific factors. The regression is then decomposed with a Shapley‑value approach to attribute portions of the total R² to four sources: scaling (compute), shared algorithmic progress, developer‑specific efficiency, and model‑specific residuals.

Key quantitative findings:

1. Scaling dominates the frontier. Across the full sample, log compute explains 32 % of performance variance; among major developers it rises to 45 %. At the performance frontier (top 5 % of MMLU‑Pro scores), 80‑90 % of the observed differences are attributable to higher compute, implying that sheer scale, not proprietary tricks, drives the latest breakthroughs. The effective compute gap between frontier models is roughly 5,000×.
2. Shared algorithmic progress is substantial. Period fixed effects indicate that achieving a given MMLU‑Pro score in 2024‑2025 requires 7.5 × less compute than in 2022‑2023. This reflects community‑wide advances such as better tokenization, optimizer tweaks, and architectural refinements that improve the compute‑to‑performance conversion rate.
3. Developer‑specific “secret sauce” matters, especially off‑frontier. Company dummies account for 14‑34 % of total variance. The estimated compute‑efficiency multipliers vary dramatically: DeepSeek is about 2.3× more efficient than the baseline “other” developers, while Microsoft’s multiplier reaches ≈ 60×. In the subset of smaller models, some firms are up to 61× more compute‑efficient than others, creating a compelling case for proprietary techniques (high‑quality data pipelines, teacher‑student training, specialized fine‑tuning).
4. Model‑specific residuals are large. Even after controlling for compute, time, and developer, the residual term explains 32‑47 % of variance. The distribution of these residuals spans a 41× gap between the 90th and 10th percentiles, indicating that within‑firm choices (architecture tweaks, hyper‑parameter search, domain‑specific fine‑tuning) can dramatically affect efficiency.

The Shapley decomposition (Figure 1) shows that for the full sample the contributions are roughly: scaling ≈ 32 %, shared progress ≈ 3‑10 %, company “secret sauce” ≈ 14‑34 %, and model‑specific factors ≈ 32‑47 %. When focusing only on major developers, scaling rises to 45 % and the secret sauce to 13‑34 %, while model effects remain sizable.

Implications:

• Compute access is the primary lever for frontier leadership. Nations or firms lacking massive compute resources are unlikely to dominate the cutting edge solely through proprietary tricks.
• Algorithmic diffusion and open research are crucial for democratization. The 7.5× efficiency gain from shared progress demonstrates the power of community‑wide innovation.
• Proprietary efficiency gains matter for cost‑effective AI. Companies that can produce smaller, high‑performing models with far less compute gain a competitive edge in markets where inference cost, energy consumption, or hardware constraints are limiting factors.
• Internal R&D efficiency is a major source of variation. The 40×+ spread in model‑specific efficiency within a single firm suggests that organizational practices, talent, and engineering pipelines are as important as the underlying algorithms.

Overall, the study provides the first robust empirical evidence that a “secret sauce” does exist in LLM development, but its impact is concentrated on improving compute efficiency for sub‑frontier models rather than pushing the absolute performance frontier. Consequently, future AI leadership will likely hinge on a combination of massive compute access, continued open algorithmic advances, and sustained investment in proprietary engineering pipelines that translate compute into performance more economically.