Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training

February 09, 2026

Reading time: 2 minute

...

📝 Original Info

Title: Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training
ArXiv ID: 2512.08894
Date: 2025-12-09
Authors: Jakub Krajewski, Amitis Shidani, Dan Busbridge, Sam Wiseman, Jason Ramapuram

📝 Abstract

While scaling laws for Large Language Models (LLMs) traditionally focus on proxy metrics like pretraining loss, predicting downstream task performance has been considered unreliable. This paper challenges that view by proposing a direct framework to model the scaling of benchmark performance from the training budget. We find that for a fixed token-toparameter ratio, a simple power law can accurately describe the scaling behavior of log accuracy on multiple popular downstream tasks. Our results show that the direct approach extrapolates better than the previously proposed two-stage procedure, which is prone to compounding errors. Furthermore, we introduce functional forms that predict accuracy across token-to-parameter ratios and account for...

📄 Full Content

Large Language Models (OpenAI et al., 2024;Team et al., 2025;DeepSeek-AI et al., 2025) based on the Transformer (Vaswani et al., 2023) architecture have achieved impressive results, approaching or exceeding human-level performance across multiple domains. Scaling laws (Hestness et al., 2017;Kaplan et al., 2020) are an established method for modeling the performance of these networks, enabling researchers to plan large-scale training runs based on curated sets of smaller experiments. Traditionally, these laws focus on predicting proxy metrics for model quality, such as pre-

…(Content truncated for length.)

📄 Read Full PDF on ArXiv