自适应跳过低重要性子层，加速长文本大模型推理
AdaSkip: Adaptive Sublayer Skipping for Accelerating Long-Context LLM Inference
- 根据上下文相似度动态判断子层重要性，决定是否跳过
- 在预填充和解码阶段均实现加速，提升推理效率
- 适合需要处理长文本的生成类大模型应用
长上下文大语言模型推理日益关键，但存储与计算成本高昂。现有逐层跳过方法在长上下文场景中存在局限：无法适应模型与上下文变化、忽略子层重要性差异、不适用于预填充阶段。本文提出 extit{AdaSkip}，一种专为长上下文推理设计的自适应子层跳过方法。该方法通过实时相似度信息识别低重要性子层，支持细粒度子层跳过，显著加速预填充与解码阶段。在多个长上下文基准与模型上进行的大量实验表明， extit{AdaSkip} 在推理性能上优于现有基线。
原文摘要 · Abstract (English)
Long-context large language models (LLMs) inference is increasingly critical, motivating a number of studies devoted to alleviating the substantial storage and computational costs in such scenarios. Layer-wise skipping methods are promising optimizations but rarely explored in long-context inference. We observe that existing layer-wise skipping strategies have several limitations when applied in long-context inference, including the inability to adapt to model and context variability, disregard for sublayer significance, and inapplicability for the prefilling phase. This paper proposes \sysname, an adaptive sublayer skipping method specifically designed for long-context inference. \sysname adaptively identifies less important layers by leveraging on-the-fly similarity information, enables sublayer-wise skipping, and accelerates both the prefilling and decoding phases. The effectiveness of \sysname is demonstrated through extensive experiments on various long-context benchmarks and models, showcasing its superior inference performance over existing baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。