Continuum: Efficient Multi-Turn LLM Agent Scheduling via KV Cache TTL
Researchers have introduced Continuum, a novel serving system designed to optimize job completion times for multi-turn Large Language Model (LLM) agent workloads. Traditional inference engines often evict Key-Value (KV) caches to maximize utilization, which disrupts efficiency in agentic workflows where LLM calls are interleaved with tool executions. Continuum addresses this by implementing a Time-to-Live (TTL) mechanism that selectively pins KV caches in GPU memory during tool calls. The TTL duration is dynamically calculated based on reload costs and potential queueing delays, ensuring robust performance despite unpredictable tool execution times. When the TTL expires, the cache is automatically evicted to free up resources. Evaluations using real-world agents like SWE-Bench and models such as Llama-3.1 and GLM-4.5 demonstrate that Continuum improves average job completion times by over eight times while simultaneously enhancing throughput. This development significantly advances the efficiency of AI agent infrastructure, addressing critical bottlenecks in GPU memory management and scheduling for complex, multi-step automated tasks.
Wire timeline
Continuum: Efficient Multi-Turn LLM Agent Scheduling via KV Cache TTL
Researchers have introduced Continuum, a novel serving system designed to optimize job completion times for multi-turn Large Language Model (LLM) agent workloads. Traditional inference engines often evict Key-Value (KV) caches to maximize utilization, which disrupts efficiency in agentic workflows where LLM calls are interleaved with tool executions. Continuum addresses this by implementing a Time-to-Live (TTL) mechanism that selectively pins KV caches in GPU memory during tool calls. The TTL duration is dynamically calculated based on reload costs and potential queueing delays, ensuring robust performance despite unpredictable tool execution times. When the TTL expires, the cache is automatically evicted to free up resources. Evaluations using real-world agents like SWE-Bench and models such as Llama-3.1 and GLM-4.5 demonstrate that Continuum improves average job completion times by over eight times while simultaneously enhancing throughput. This development significantly advances the efficiency of AI agent infrastructure, addressing critical bottlenecks in GPU memory management and scheduling for complex, multi-step automated tasks.
cs.AI updates on arXiv.org