用动态三角注意力提升大模型长文本推理效率
Ltri-LLM: Streaming Long Context Inference for LLMs with Training-Free Dynamic Triangular Attention Pattern
- 将键值对分段存储，按需检索相关上下文
- 在多个长文本任务上接近全注意力性能
- 无需训练，适合需要高效流式处理的场景
当前大语言模型的注意力机制具有二次计算复杂度，导致长文本推理成本高昂。现有方法通过键值压缩或稀疏注意力保留关键上下文，实现近乎无限长度的流式处理，但在检索任务中性能仍远低于全注意力（FA）。本文分析发现，大模型注意力分布具有强局部相关性，天然体现输入上下文的分块机制。为此提出Ltri-LLM框架：将键值对分段存储于离线索引，查询时仅召回相关片段。在多个主流长文本基准测试中，Ltri-LLM在保持流式推理效率的同时，性能接近全注意力。
原文摘要 · Abstract (English)
The quadratic computational complexity of the attention mechanism in current Large Language Models (LLMs) renders inference with long contexts prohibitively expensive. To address this challenge, various approaches aim to retain critical portions of the context to optimally approximate Full Attention (FA) through Key-Value (KV) compression or Sparse Attention (SA), enabling the processing of virtually unlimited text lengths in a streaming manner. However, these methods struggle to achieve performance levels comparable to FA, particularly in retrieval tasks. In this paper, our analysis of attention head patterns reveals that LLMs' attention distributions show strong local correlations, naturally reflecting a chunking mechanism for input context. We propose Ltri-LLM framework, which divides KVs into spans, stores them in an offline index, and retrieves the relevant KVs into memory for various queries. Experimental results on popular long text benchmarks show that Ltri-LLM can achieve performance close to FA while maintaining efficient, streaming-based inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。