轻量改造大模型，让长文本生成更快更省显存
LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptation
- 识别模型中处理新旧信息的'懒惰层'，用流式注意力替代全注意力
- 无需训练即可提速2.17倍，性能损失小于1.5%，数学推理达53.3%
- 适合想快速提升长文本生成效率的开发者和研究者
将语言模型扩展至更长上下文会因键值缓存成本上升带来巨大内存压力。受混合模型效率启发，并利用预训练大模型的广泛可用性，我们探索将Transformer模型转化为更高效的混合架构。本文提出LightTransfer，一种轻量级方法，可将LLaMA等模型转变为混合变体。该方法识别出聚焦于近期或初始标记的‘懒惰层’，并将其完整注意力替换为流式注意力。此转换无需训练即可用于长上下文理解任务，或仅需少量微调即可用于需要强推理能力的o1类长推理生成任务。在多个基准和模型（如LLaMA、Mistral、QwQ-STILL）上的实验表明，即使一半层被识别为懒惰层，LightTransfer仍可实现高达2.17倍的吞吐量提升，性能损失低于1.5%（LongBench），并在数学基准AIME24上达到53.3%的得分，接近先进o1类模型的表现。
原文摘要 · Abstract (English)
Scaling language models to handle longer contexts introduces substantial memory challenges due to the growing cost of key-value (KV) caches. Motivated by the efficiency gains of hybrid models and the broad availability of pretrained large transformer backbones, we explore transitioning transformer models into hybrid architectures for a more efficient generation. In this work, we propose LightTransfer, a lightweight method that transforms models such as LLaMA into hybrid variants. Our approach identifies lazy layers -- those focusing on recent or initial tokens -- and replaces their full attention with streaming attention. This transformation can be performed without any training for long-context understanding tasks or with minimal fine-tuning for o1-like long reasoning generation tasks that require stronger reasoning capabilities. Experiments across diverse benchmarks and models (e.g., LLaMA, Mistral, QwQ-STILL) demonstrate that, even when half of the layers are identified as lazy, LightTransfer achieves up to 2.17$\times$ throughput improvement with minimal performance loss ($<1.5\%$ on LongBench) and achieves 53.3\% on math benchmark AIME24 of advanced o1-like long reasoning model QwQ-STILL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。