SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference
Researchers have introduced SPECTRE, a novel serving framework designed to enhance the efficiency of Large Language Model (LLM) inference in multi-model cloud systems. Addressing the long-tailed demand distribution where popular large models are heavily loaded while smaller tail models remain underutilized, SPECTRE repurposes these idle tail-model services as remote drafters for speculative decoding. The framework enables parallel draft generation and target-side verification, utilizing three key techniques: a hybrid ordinary-parallel speculative decoding strategy, speculative priority scheduling to maintain overlap under multi-tenant traffic, and draft-side prompt compression to reduce latency. Implemented within the SGLang platform, SPECTRE was evaluated across various model pairs and workloads. Results demonstrate significant performance gains, achieving up to a 2.28x speedup over standard autoregressive decoding and a 66% relative improvement over leading speculative decoding baselines in large-model deployments like Qwen3-235B-A22B. Crucially, this throughput boost for large models incurs only minor interference with the native workloads of the tail-model services, offering a resource-efficient solution for scalable LLM serving infrastructure.
Wire timeline
SPECTRE: Hybrid Ordinary-Parallel Speculative Serving for Resource-Efficient LLM Inference
Researchers have introduced SPECTRE, a novel serving framework designed to enhance the efficiency of Large Language Model (LLM) inference in multi-model cloud systems. Addressing the long-tailed demand distribution where popular large models are heavily loaded while smaller tail models remain underutilized, SPECTRE repurposes these idle tail-model services as remote drafters for speculative decoding. The framework enables parallel draft generation and target-side verification, utilizing three key techniques: a hybrid ordinary-parallel speculative decoding strategy, speculative priority scheduling to maintain overlap under multi-tenant traffic, and draft-side prompt compression to reduce latency. Implemented within the SGLang platform, SPECTRE was evaluated across various model pairs and workloads. Results demonstrate significant performance gains, achieving up to a 2.28x speedup over standard autoregressive decoding and a 66% relative improvement over leading speculative decoding baselines in large-model deployments like Qwen3-235B-A22B. Crucially, this throughput boost for large models incurs only minor interference with the native workloads of the tail-model services, offering a resource-efficient solution for scalable LLM serving infrastructure.
cs.AI updates on arXiv.org