通过动态跳过部分层加速大模型推理，性能几乎无损且内存减少一半。
LoRA-Drop: Temporal LoRA Decoding for Efficient LLM Inference
- 用低秩修正替代部分层计算，定期刷新防止误差累积。
- 最高提速2.6倍，KV缓存减少45%~55%，准确率损失小于0.5个百分点。
- 无需额外路由网络，适合部署在现有系统中，提升推理效率。
自回归大语言模型的推理受限于逐词生成带来的顺序计算开销，每次生成新词通常需执行全部Transformer层。现有动态深度与层跳过方法虽能降低计算量，但常依赖辅助路由机制或在跳过层未补偿时导致精度下降。本文提出 extbf{LoRA-Drop}，一种即插即用的推理框架，通过为固定子集中间层施加 extit{时间计算调度}：多数解码步骤中，选定层复用前一词隐藏状态并应用低秩LoRA修正；周期性 extit{刷新}步骤则完整运行模型以防止误差漂移。该方法无需路由网络，兼容标准键值缓存（KV caching），可通过跳过可丢弃层的缓存更新进一步缩减缓存占用，并在 extbf{LLaMA2-7B}、 extbf{LLaMA3-8B}、 extbf{Qwen2.5-7B}和 extbf{Qwen2.5-14B}上实现最高 extbf{2.6×}的解码加速与 extbf{45–55 extbackslash extbackslash%}的缓存压缩，同时保持精度损失低于 extbf{0.5个百分点（pp）}。在推理（GSM8K、MATH、BBH）、代码生成（HumanEval、MBPP）及长文本/多语言基准（LongBench、XNLI、XCOPA）上的评估表明，存在一个稳定的调度配置区间，在维持质量的同时实现显著效率提升，为大模型自适应容量推理提供简单路径。
原文摘要 · Abstract (English)
Autoregressive large language models (LLMs) are bottlenecked by sequential decoding, where each new token typically requires executing all transformer layers. Existing dynamic-depth and layer-skipping methods reduce this cost, but often rely on auxiliary routing mechanisms or incur accuracy degradation when bypassed layers are left uncompensated. We present \textbf{LoRA-Drop}, a plug-and-play inference framework that accelerates decoding by applying a \emph{temporal compute schedule} to a fixed subset of intermediate layers: on most decoding steps, selected layers reuse the previous-token hidden state and apply a low-rank LoRA correction, while periodic \emph{refresh} steps execute the full model to prevent drift. LoRA-Drop requires no routing network, is compatible with standard KV caching, and can reduce KV-cache footprint by skipping KV updates in droppable layers during LoRA steps and refreshing periodically. Across \textbf{LLaMA2-7B}, \textbf{LLaMA3-8B}, \textbf{Qwen2.5-7B}, and \textbf{Qwen2.5-14B}, LoRA-Drop achieves up to \textbf{2.6$\times$ faster decoding} and \textbf{45--55\% KV-cache reduction} while staying within \textbf{0.5 percentage points (pp)} of baseline accuracy. Evaluations on reasoning (GSM8K, MATH, BBH), code generation (HumanEval, MBPP), and long-context/multilingual benchmarks (LongBench, XNLI, XCOPA) identify a consistent \emph{safe zone} of scheduling configurations that preserves quality while delivering substantial efficiency gains, providing a simple path toward adaptive-capacity inference in LLMs. Codes are available at https://github.com/hosseinbv/LoRA-Drop.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。