AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving

📝 Original Info

• Title: AdaptCache: KV Cache Native Storage Hierarchy for Low-Delay and High-Quality Language Model Serving
• ArXiv ID: 2509.00105
• Date: 2025-08-28
• Authors: ** 논문에 명시된 저자 정보가 제공되지 않았습니다. (필요 시 원문 PDF 혹은 학회 페이지에서 확인 바랍니다.) **

📝 Abstract

Large language model (LLM) applications often reuse previously processed context, such as chat history and documents, which introduces significant redundant computation. Existing LLM serving systems address such redundant computation by storing the KV caches of processed context and loading the corresponding KV cache when a new request reuses the context. Further, as these LLM applications scale, the total size of KV caches becomes excessively large and requires both DRAM and SSD for full storage. However, prior work that stores KV caches in DRAM and SSD suffers from high loading delays, as most KV cache hits come from SSD, which is slow to load. To increase the KV cache hit rate on DRAM, we identify lossy KV cache compression as a promising approach. We design a lossy compression system that decides the compression algorithm, compression rate and device placement for each KV cache entry to maximise DRAM hits and minimise loading delay without significantly degrading generation quality. Compared to various static compression baselines across three tasks, our system AdaptCache achieves 1.43--2.4 x delay savings at the same quality and 6--55% quality improvements at the same delay.

💡 Deep Analysis

📄 Full Content

Reference