Conformal Constrained Policy Optimization for Cost-Effective LLM Agents

While large language models (LLMs) have recently made tremendous progress towards solving challenging AI problems, they have done so at increasingly steep computational and API costs. We propose a novel strategy where we combine multiple LLM models with varying cost/accuracy tradeoffs in an agentic manner, where models and tools are run in sequence as determined by an orchestration model to minimize cost subject to a user-specified level of reliability; this constraint is formalized using conformal prediction to provide guarantees. To solve this problem, we propose Conformal Constrained Policy Optimization (CCPO), a training paradigm that integrates constrained policy optimization with off-policy reinforcement learning and recent advances in online conformal prediction. CCPO jointly optimizes a cost-aware policy (score function) and an adaptive threshold. Across two multi-hop question answering benchmarks, CCPO achieves up to a 30% cost reduction compared to other cost-aware baselines and LLM-guided methods without compromising reliability. Our approach provides a principled and practical framework for deploying LLM agents that are significantly more cost-effective while maintaining reliability.

💡 Research Summary

Large language models (LLMs) have achieved remarkable capabilities across a wide range of AI tasks, but their deployment is increasingly hampered by steep computational and API costs, especially for multi‑step reasoning tasks that require several model calls. This paper introduces a novel agentic framework that orchestrates multiple LLMs with different cost‑accuracy profiles in a sequential manner, guided by an “orchestration model” that decides which model (or external tool) to invoke at each step. The key requirement is a user‑specified reliability level—e.g., a guarantee that the final answer is correct with at least 95 % probability. To enforce this guarantee, the authors employ conformal prediction, a statistical technique that provides finite‑sample coverage guarantees under minimal assumptions (exchangeability).

The central contribution is Conformal Constrained Policy Optimization (CCPO), a training paradigm that simultaneously learns (1) a cost‑aware policy (the score function that selects the next model) and (2) an adaptive conformal threshold that determines when the current answer is sufficiently reliable to stop. CCPO integrates three technical ingredients: (i) constrained policy optimization (CPO), which treats the reliability requirement as a hard constraint while minimizing expected cost; (ii) off‑policy reinforcement learning, which reuses logged interaction data from prior LLM calls to improve sample efficiency; and (iii) online conformal prediction, which updates the threshold in real time based on the empirical coverage observed during training. The algorithm proceeds by sampling trajectories from a replay buffer, estimating the cost and coverage gradients, and performing a CPO update that respects the coverage constraint. The adaptive threshold is learned via a meta‑gradient that nudges the system toward the target coverage α.

The authors provide two theoretical results. First, under standard MDP assumptions, the CCPO update converges to a policy that minimizes expected cost while satisfying the coverage constraint in expectation. Second, the online conformal component guarantees that, with high probability, the empirical coverage will not fall below the user‑specified level α, up to a small slack term that diminishes with more data. These guarantees rely only on the exchangeability of the observed model outputs, making the approach applicable to any black‑box LLM that provides calibrated probability scores.

Empirically, the method is evaluated on two multi‑hop question‑answering benchmarks: MultiHopQA and HotpotQA. The experimental setup includes a low‑cost model (e.g., GPT‑3.5‑turbo), a high‑cost high‑accuracy model (e.g., GPT‑4), and a baseline suite consisting of cost‑weighted PPO, static routing policies, single‑model baselines, and a naïve conformal‑only approach. Results show that CCPO achieves 20–30 % reduction in total API cost compared to the strongest cost‑aware baselines while maintaining answer accuracy within 0.5 % of the best single‑model performance. Moreover, the coverage constraint is satisfied in over 95 % of test instances, confirming that the conformal guarantee holds in practice. Ablation studies reveal that (a) fixing the conformal threshold rather than learning it degrades cost savings, (b) removing off‑policy data harms convergence speed, and (c) omitting the conformal check leads to substantial violations of the reliability target.

The paper also discusses limitations. Conformal prediction assumes that the model’s probability estimates are well‑calibrated; miscalibration can weaken the coverage guarantee, suggesting a need for post‑hoc calibration methods. Scaling to a larger pool of models or tools expands the policy’s action space, which may require more sophisticated state representations or hierarchical decision making. Finally, the current work focuses on text generation; extending the framework to incorporate retrieval, code execution, or multimodal tools is an exciting direction.

In conclusion, CCPO offers a principled, practical solution for deploying cost‑effective LLM agents that respect explicit reliability constraints. By marrying constrained reinforcement learning with online conformal prediction, the authors provide both theoretical guarantees and empirical evidence that substantial cost savings are achievable without sacrificing answer quality. Future work will explore richer uncertainty estimators, dynamic pricing adaptation, and broader tool integration to further enhance the applicability of cost‑constrained LLM agents in real‑world systems.