Curriculum-Guided Layer Scaling for Language Model Pretraining

As the cost of pretraining large language models grows, there is continued interest in strategies to improve learning efficiency during this core training stage. Motivated by cognitive development, where humans gradually build knowledge as their brains mature, we propose Curriculum-Guided Layer Scaling (CGLS), a framework for compute-efficient pretraining that synchronizes increasing data difficulty with model growth through progressive layer stacking (i.e. gradually adding layers during training). At the 100M parameter scale, using a curriculum transitioning from synthetic short stories to general web data, CGLS outperforms baseline methods on the question-answering benchmarks PIQA and ARC. Pretraining at the 1.2B scale, we stratify the DataComp-LM corpus with a DistilBERT-based classifier and progress from general text to highly technical or specialized content. Our results show that progressively increasing model depth alongside sample difficulty leads to better generalization and zero-shot performance on various downstream benchmarks. Altogether, our findings demonstrate that CGLS unlocks the potential of progressive stacking, offering a simple yet effective strategy for improving generalization on knowledge-intensive and reasoning tasks.

💡 Research Summary

The paper “Curriculum-Guided Layer Scaling for Language Model Pretraining” introduces a novel framework designed to improve the compute efficiency and final performance of large language model (LLM) pretraining by synchronizing the growth of model capacity with the increasing complexity of training data. Motivated by human cognitive development, where learning progresses from simple to complex concepts in tandem with brain maturation, the proposed method, Curriculum-Guided Layer Scaling (CGLS), addresses the high cost of standard single-pass pretraining.

CGLS operates on two coordinated axes. First, it employs a data curriculum that gradually transitions the training distribution from easier, more structured samples to harder, more diverse and specialized content. Second, it progressively expands the model’s architectural depth by adding new transformer layers at specific training stages. A key innovation is the staged training process when adding layers: the newly added, randomly initialized layers are first trained in isolation on top of the frozen existing model, before the entire expanded model is fine-tuned. This protects previously learned representations during capacity increases.

The authors rigorously evaluate CGLS at two scales: a 100M parameter model (GPT-2 Small scale) and a 1.2B parameter model (LLaMA-3.2-1B scale). For the smaller model, they use an explicit curriculum moving from synthetic short stories (TinyStories) to general web text (DataComp-LM). For the larger model, they demonstrate scalability by creating a curriculum within a single large corpus (DataComp-LM). They use a DistilBERT classifier, trained on a small set of GPT-4o-labeled samples, to stratify documents into three complexity levels: “High School,” “Undergraduate,” and “Graduate/Advanced.” The training then progresses from a mix favoring simpler documents to one dominated by complex, technical content.

All experiments are compute-controlled, meaning competing methods are compared under an identical total FLOPs budget. The results show that CGLS consistently outperforms standard pretraining (with a fixed architecture) and other progressive stacking baselines. Performance gains are particularly notable on knowledge-intensive and reasoning-heavy downstream tasks evaluated in a zero-shot setting, such as ARC (AI2 Reasoning Challenge) and PIQA (Physical Interaction QA). At the 1.2B scale trained on 2.5B tokens, CGLS achieved an average improvement of 1.70% across multiple benchmarks, with gains up to 5% on ARC-Easy. When scaled to a Chinchilla-optimal 20B tokens, the average improvement grew to 3.90%.

The paper’s central finding is that jointly scaling model depth and data complexity unlocks the potential of progressive stacking, which alone can underperform on knowledge tasks. By providing an appropriately paced learning signal that matches the model’s growing representational capacity, CGLS leads to more efficient learning and better generalization. The work provides both a practical framework for efficient pretraining and evidence supporting a developmental, curriculum-based approach to building machine intelligence.