What is the Role of Small Models in the LLM Era: A Survey

Large Language Models (LLMs) have made significant progress in advancing artificial general intelligence (AGI), leading to the development of increasingly large models such as GPT-4 and LLaMA-405B. However, scaling up model sizes results in exponentially higher computational costs and energy consumption, making these models impractical for academic researchers and businesses with limited resources. At the same time, Small Models (SMs) are frequently used in practical settings, although their significance is currently underestimated. This raises important questions about the role of small models in the era of LLMs, a topic that has received limited attention in prior research. In this work, we systematically examine the relationship between LLMs and SMs from two key perspectives: Collaboration and Competition. We hope this survey provides valuable insights for practitioners, fostering a deeper understanding of the contribution of small models and promoting more efficient use of computational resources. The code is available at https://github.com/tigerchen52/role_of_small_models

💡 Research Summary

**
The paper surveys the evolving role of Small Models (SMs) in the era dominated by Large Language Models (LLMs). While LLMs such as GPT‑4 and LLaMA‑405B achieve state‑of‑the‑art performance across a wide range of tasks, their training and inference costs, energy consumption, and deployment constraints make them impractical for many academic and commercial settings. In contrast, open‑source models with fewer than one billion parameters—e.g., BERT‑base, RoBERTa‑large, and various decoder‑only models—remain highly popular, as evidenced by download statistics from HuggingFace.

The authors first clarify what constitutes a “small model.” Rather than a fixed parameter threshold, they adopt a relative definition: a model is small when it has considerably fewer parameters than the LLM it is compared against. The survey focuses primarily on transformer‑based models under 1 B parameters, but acknowledges that earlier shallow networks or statistical models also fall under the SM umbrella.

A comparative table highlights four dimensions: accuracy, generality, efficiency, and interpretability. LLMs excel in accuracy and generality but are resource‑intensive and opaque. SMs lag in raw performance but can be competitive when enhanced by knowledge distillation, fine‑tuning on domain‑specific data, or prompt engineering. Their low computational footprint makes them suitable for real‑time services, edge devices, and regulated industries where model transparency is essential.

The core contribution is a two‑pronged analysis of SM–LLM interaction: Collaboration and Competition/Complementarity.

Collaboration is examined through the lifecycle of an LLM: data preparation, inference, and evaluation.
Data preparation: SMs act as cheap proxy classifiers for data selection (filtering noisy, toxic, duplicated, or private content) and data re‑weighting (assigning domain‑level or instance‑level importance scores). Techniques such as DoReMi, AutoScale, and PRESENCE demonstrate that a 280 M‑parameter proxy can guide the training of an 8 B model, achieving faster convergence and higher downstream accuracy.
Inference: SMs enable speculative decoding, model routing, and cascading. In speculative decoding, a small model generates candidate tokens quickly, while the LLM validates them, reducing overall FLOPs. Model routing dynamically selects the most appropriate SM or LLM based on input characteristics, minimizing latency. Retrieval‑augmented generation and domain adaptation further allow SMs to supply external knowledge or specialized expertise, enhancing LLM reasoning without incurring full‑scale computation.
Evaluation: Lightweight SMs serve as verifiers that assess factuality, toxicity, and bias of LLM outputs. When discrepancies are detected, they trigger prompt repair or re‑generation, improving reliability at minimal cost.

Competition and Complementarity emphasizes the intrinsic advantages of SMs: simplicity, lower cost, and higher interpretability. Knowledge and representation distillation techniques transfer LLM knowledge into SMs, enabling sub‑billion‑parameter models to approach state‑of‑the‑art performance on specific benchmarks. In regulated sectors such as healthcare, finance, and law, the demand for transparent, auditable models gives SMs a decisive edge.

The survey also reviews related work, noting that previous reviews have catalogued SM architectures, training tricks, and compression methods but have largely treated SMs in isolation. By contrast, this paper situates SMs within the broader AI ecosystem, focusing on their synergistic and competitive dynamics with LLMs.

Future research directions identified include: advanced pruning and quantization for further SM compression; automated data re‑weighting via reinforcement learning; multimodal SMs that collaborate with LLMs for vision‑language tasks; and the design of “model hubs” that orchestrate dynamic switching between SMs and LLMs across cloud and edge environments.

In conclusion, the authors argue that the AI landscape will shift from a monolithic LLM‑centric paradigm to a hybrid architecture where small and large models coexist, each leveraged for its strengths. The survey provides actionable guidelines for practitioners facing resource constraints, and it underscores the enduring relevance of small models as both cost‑effective tools and essential components of a sustainable, interpretable AI future.