Radar动态筛选关键上下文token，加速Transformer长文本推理。
Radar: Fast Long-Context Decoding for Any Transformer
- 不需训练，动态搜索重要上下文token以减少计算量。
- 在多种架构上实现最优性能，时间复杂度显著降低。
- 适合需要高效处理长文本的场景，如文档理解、代码生成。
Transformer模型在诸多任务中表现出色，但其点积注意力机制随上下文长度增长呈二次方时间开销，难以扩展至长序列。本文提出Radar，一种无需训练的方法，在推理时动态搜索最具重要性的上下文token，从而降低解码时间复杂度。该方法适用于任意预训练Transformer，无需启发式丢弃token或重新训练。我们提供了理论支持，证明Radar能以高概率准确识别关键token。在多种任务上的广泛实验表明，Radar在不同架构下均达到当前最佳性能，同时显著降低时间复杂度，为Transformer的长上下文高效处理提供了实用方案。
原文摘要 · Abstract (English)
Transformer models have demonstrated exceptional performance across a wide range of applications. Though forming the foundation of Transformer models, the dot-product attention does not scale well to long-context data since its time requirement grows quadratically with context length. In this work, we propose Radar, a training-free approach that accelerates inference by dynamically searching for the most important context tokens. For any pre-trained Transformer, Radar can reduce the decoding time complexity without training or heuristically evicting tokens. Moreover, we provide theoretical justification for our approach, demonstrating that Radar can reliably identify the most important tokens with high probability. We conduct extensive comparisons with the previous methods on a wide range of tasks. The results demonstrate that Radar achieves the state-of-the-art performance across different architectures with reduced time complexity, offering a practical solution for efficient long-context processing of Transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。