FastTTS: Accelerating Test-Time Scaling for Edge LLM Reasoning

Recent advances in reasoning Large Language Models (LLMs) are driving the emergence of agentic AI systems. Edge deployment of LLM agents near end users is increasingly necessary to protect data privacy, enable offline use, and provide responsive interaction with local context. However, strict memory constraints on edge devices limit deployment to smaller LLMs, whose reasoning capabilities are much weaker than those of large cloud models, hindering practical deployment of edge agentic AI. Test-Time Scaling (TTS) offers a promising solution by allocating more compute during inference to enhance the reasoning capability of edge LLMs. However, current TTS methods introduce heavy hardware performance overhead on resource-constrained devices, making them impractical for real applications. To address this challenge, we present FastTTS, a serving system that enables fast and efficient TTS for memory-constrained LLM reasoning. After analyzing common patterns across various TTS methods and identifying their performance bottlenecks, we introduce three novel techniques: i) Speculative Beam Extension, which mitigates system stragglers caused by irregular reasoning paths, ii) Asymmetric Multi-Model Memory Allocation, which dynamically balances memory usage between token generation and reasoning-step verification, and iii) Dynamic Prefix-Aware Scheduling, which optimizes reasoning execution to maximize KV-cache reuse across search paths. FastTTS offers a plug-and-play third-party library on top of vLLM, enabling edge LLMs on a single consumer GPU to match cloud-model accuracy and cloud-measured latency. Comprehensive evaluation shows that FastTTS achieves an average 2.2x higher goodput and reduces latency by 38%–68% compared to the vLLM baseline; it pushes the boundaries of low-latency TTS on memory-constrained edge devices and highlights the potential for democratizing agentic AI.

💡 Research Summary

**
The paper “FastTTS: Accelerating Test‑Time Scaling for Edge LLM Reasoning” tackles the problem of deploying strong reasoning large language models (LLMs) on memory‑constrained edge devices. While recent advances in LLM reasoning have enabled powerful agentic AI systems, edge hardware (typically a single consumer‑grade GPU with 8‑24 GB VRAM) can only host small models (≤ 7 B parameters), whose reasoning ability lags far behind cloud‑scale models. Test‑Time Scaling (TTS) – allocating extra compute at inference time – promises to bridge this gap, but existing TTS implementations suffer from severe performance penalties on edge hardware due to irregular workloads, inefficient KV‑cache reuse, and the need to keep both a generator and a verifier model in memory.

FastTTS is a serving system built on top of the popular vLLM framework that makes TTS practical for edge deployment. The authors first abstract the common execution pattern of modern verifier‑guided TTS methods as a two‑stage loop: (1) Generation – each active beam extends its reasoning step by generating a variable‑length token sequence, and (2) Verification – a verifier (or process‑reward model) scores the new step, after which top‑scoring beams are kept and the rest pruned. Profiling this pattern reveals three core challenges:

1. Hardware under‑utilization from irregular work‑loads – because different beams generate different numbers of tokens, the system must wait for the longest “straggler” before proceeding, leaving the GPU idle for a large fraction of time.
2. Suboptimal exploitation of dynamic prefix sharing – many beams share common prefixes (early “thinking” steps). Existing schedulers ignore this dynamic locality, causing frequent KV‑cache evictions and costly recomputation, especially problematic under tight memory budgets.
3. Constrained memory for multi‑model execution – keeping both generator and verifier in GPU memory forces small batch sizes, limiting the throughput gains that TTS could otherwise provide.

To address these, FastTTS introduces three synergistic techniques:

• Speculative Beam Extension – anticipates the longest possible generation length for each beam and speculatively launches token generation ahead of the verification barrier. By overlapping work and keeping the GPU busy even while shorter beams finish, the approach eliminates most of the straggler effect. The implementation uses asynchronous pipelines and dynamic workload balancing, achieving >70 % GPU utilization throughout the generation phase.

• Asymmetric Multi‑Model Memory Allocation – rather than statically partitioning GPU memory equally between generator and verifier, FastTTS monitors per‑stage memory demand and reallocates memory on‑the‑fly. During generation, most of the memory is given to the generator; during verification, it is shifted to the verifier. This dynamic, asymmetric allocation enables two models (e.g., a 7 B generator and a 2 B verifier) to coexist on a 24 GB GPU, increasing effective batch size by up to 1.8× and reducing latency.

• Dynamic Prefix‑Aware Scheduling – tracks which beams share identical prefixes and reorders execution to maximize KV‑cache reuse. By grouping beams with common prefixes together, the system avoids unnecessary cache evictions, cutting cache‑misses by roughly 45 % and saving recomputation time. The scheduler adapts at runtime, handling the highly dynamic branching factor of modern TTS algorithms (e.g., diverse selection, dynamic branching).

FastTTS is delivered as a plug‑and‑play library that requires only a few lines of code to replace the default vLLM scheduler. The authors evaluate the system on several reasoning benchmarks (AIME, MATH‑500, MA‑TH) using Qwen2.5‑Math‑1.5B as the edge model and compare against a vanilla vLLM baseline. Results show:

• Goodput – an average 2.2× increase over baseline, meaning more useful reasoning results per unit time.
• Latency – reductions ranging from 38 % to 68 % across tasks; for complex multi‑step problems latency drops from ~200 s (baseline) to 80‑120 s with FastTTS.
• GPU Utilization – generation phase utilization rises from ~30 % (baseline) to >85 % with speculative beam extension; verification phase stays near 95 %.
• Memory Efficiency – despite doubling the effective memory footprint (6 GB → 12 GB) due to dual‑model loading, the asymmetric allocator keeps the system within the 24 GB limit while still improving throughput.

The paper also demonstrates that FastTTS works across a variety of TTS strategies (standard beam search, diverse selection, dynamic branching), consistently narrowing the accuracy‑latency gap between edge and cloud models. The authors argue that FastTTS paves the way for democratized agentic AI, where powerful reasoning agents can run locally on user devices, preserving privacy and enabling offline operation in sensitive domains such as healthcare, autonomous driving, and defense.

In the discussion, the authors outline future directions: extending the system to support more than two cooperating models (e.g., tool‑use models), integrating with specialized accelerators (NPU, FPGA) for KV‑cache storage, and exploring adaptive precision techniques to further reduce memory while preserving accuracy. Overall, FastTTS represents a significant step toward making high‑quality LLM reasoning feasible on edge hardware without sacrificing the performance gains promised by test‑time scaling.