通过激活LoRA实现跨模型缓存复用,大幅降低多适配器推理延迟。
Efficient Multi-Adapter LLM Serving via Cross-Model KV-Cache Reuse with Activated LoRA
- 基于aLoRA设计跨模型键值缓存复用机制,支持细粒度适配器切换。
- 在多轮任务中实现最高58倍端到端延迟降低,首次令牌时间提升超100倍。
- 适合需要动态切换多个任务适配器的生产级大模型服务场景。
现代大语言模型系统越来越多依赖由多个任务特定适配器组成的多轮处理流程,但现有服务框架效率低下,在适配器间切换时产生大量重复计算开销。本文提出首个支持基模型与适配模型间前缀缓存复用的LLM服务引擎,基于激活LoRA(aLoRA),实现推理过程中的高效、细粒度适配器切换。设计上扩展vLLM框架,引入基线对齐的块哈希与执行路径中的激活感知掩码,使缓存可在不同模型间复用,同时保持与现有服务优化兼容。集成至生产级推理栈后,该方法支持动态适配器激活,避免过度的键值张量重计算。在代表性多轮、多适配器流水线上的评估表明,相比标准LoRA基线,端到端延迟最高降低58倍,首次令牌时间提升超过100倍,性能优势随模型规模和序列长度增长,并贯穿请求生命周期所有阶段。本工作将参数高效模型适配与高性能服务相结合,首次完整实现了现代大模型推理引擎中的跨模型键值缓存复用。
原文摘要 · Abstract (English)
Modern large language model (LLM) systems increasingly rely on multi-turn pipelines that are composed of multiple task-specific adapters, yet existing serving frameworks remain inefficient, incurring substantial recomputation overhead when switching between adapters. We present the first LLM serving engine that supports cross-model prefix cache reuse between base and adapted models via Activated LoRA (aLoRA), enabling efficient and fine-grained adapter switching during inference. Our design extends the vLLM framework by introducing base-aligned block hashing and activation-aware masking within the model execution path, permitting cache reuse across models while preserving compatibility with existing serving engine optimizations. Integrated into a production-grade inference stack, this approach supports dynamic adapter activation without excessive key-value tensor recomputation. Evaluation across representative multi-turn, multi-adapter pipelines demonstrates up to 58x end-to-end latency reduction and over 100x time-to-first-token improvement relative to standard LoRA baselines, with benefits that scale with model size and sequence length and manifest across all stages of the request lifecycle. This work bridges parameter-efficient model adaptation with high-performance serving, providing the first complete realization of cross-model KV-cache reuse in modern LLM inference engines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。