Balancing Sustainability And Performance: The Role Of Small-Scale LLMs In Agentic Artificial Intelligence Systems

As large language models become integral to agentic artificial intelligence systems, their energy demands during inference may pose significant sustainability challenges. This study investigates whether deploying smaller-scale language models can reduce energy consumption without compromising responsiveness and output quality in a multi-agent, real-world environments. We conduct a comparative analysis across language models of varying scales to quantify trade-offs between efficiency and performance. Results show that smaller open-weights models can lower energy usage while preserving task quality. Building on these findings, we propose practical guidelines for sustainable artificial intelligence design, including optimal batch size configuration and computation resource allocation. These insights offer actionable strategies for developing scalable, environmentally responsible artificial intelligence systems.

💡 Research Summary

The paper addresses the growing sustainability concerns associated with the inference phase of large language models (LLMs) that power agentic artificial intelligence systems. While LLMs enable sophisticated multi‑agent interactions, their energy consumption during real‑time inference can be substantial, raising both environmental and cost issues. To explore whether smaller, open‑weight LLMs can mitigate these problems without sacrificing performance, the authors conduct a comprehensive benchmark that simultaneously evaluates three dimensions: (1) environmental impact measured as average GPU energy consumption per request (joules), (2) user experience captured by decode latency (the time needed to generate the response after prompt processing), and (3) output quality assessed through a two‑pronged approach—Macro F1 score on a “grounded/ungrounded/small‑talk” classification task and an LLM‑as‑a‑Judge rubric that scores factual correctness and relevance on a 0‑1 scale.

The experimental setup mirrors a real‑world multi‑agent deployment. The authors extract 1,000 conversation turns from a production system, each averaging 8,000 tokens (max 25,500). The reference system uses the closed‑source GPT‑4o model via API, producing responses of about 66 tokens (max 118). The benchmark runs on the ML‑Energy framework, which records per‑request energy and latency on identical GPU hardware.

A diverse set of models is evaluated: the Qwen 2.5 family (0.5 B to 72 B parameters), several compressed variants of the 7 B model (4‑bit and 8‑bit quantization, activation‑aware weight quantization, and knowledge‑distilled students at 1.5 B and 7 B), plus 20 additional open‑source LLMs spanning Gemma, Mistral, Falcon, Phi‑4, and a MoE Llama‑Scout. In total, 28 base models and 7 compressed versions are examined.

Key findings:

• Energy – Energy per request scales roughly linearly with parameter count. Models ≤ 7 B consume 30‑45 % less energy than GPT‑4o; the 0.5 B model saves over 55 %.
• Latency – Larger models exhibit higher decode latency, but increasing batch size (8‑16) reduces latency more effectively than model compression alone. For example, a 14 B model with batch‑size 16 shows a 20 % latency drop.
• Quality – The 7 B Qwen‑Instruct achieves a Macro F1 of ≈ 0.92, essentially matching GPT‑4o, while the 0.5 B model records ≈ 0.86, still acceptable for many applications. Quantized 4‑bit models cut energy further but lose 0.05‑0.08 in quality scores. Knowledge‑distilled students retain near‑original quality (≈ 0.02 loss) while reducing energy and latency by > 20 %.

To synthesize these dimensions, the authors propose an Overall Metric (OM):
OM = w₁·(Quality_E/Quality_R)² + w₂·(Latency_R/Latency_E)² + w₃·(Energy_R/Energy_E)²,
with weights w₁ = 0.5, w₂ = 0.3, w₃ = 0.2 reflecting typical priority hierarchies. Under this formulation, the 7 B Qwen‑Instruct and its distilled 7 B counterpart achieve OM > 1, indicating superior overall performance to the proprietary baseline.

The paper concludes that carefully selected small‑scale open‑weight LLMs, combined with appropriate compression (quantization or distillation) and batch‑size tuning, can replace large closed‑source models in production agents while delivering meaningful energy savings, lower latency, and comparable output quality. Practical guidelines are offered for model selection, batch configuration, and weighting of sustainability versus performance objectives, providing enterprises with a concrete roadmap to greener, cost‑effective AI deployments.