通过压缩查询与键向量，高效选择关键词元以扩展上下文长度。
Unshackling Context Length: An Efficient Selective Attention Approach through Query-Key Compression
- 压缩查询和键向量至低维表示，实现细粒度词元筛选。
- 在最长256k的序列上表现优于其他选择性注意力方法。
- 适合需要多信息召回的任务，尤其适用于长文本理解场景。
长序列处理仍是大语言模型的核心挑战。现有序列外推中的词元选择方法或采用固定淘汰策略，或按块选取，可能丢失关键信息。我们提出高效选择性注意力（ESA），通过在词元层面筛选最关键的词元计算注意力，从而扩展上下文长度。ESA通过将查询和键向量压缩为低维表示，降低词元选择的计算复杂度。我们在最大长度达256k的长序列基准上，使用上下文长度分别为8k和32k的开源LLM进行评估。ESA在需多信息检索的任务中显著优于其他选择性注意力方法，在多种任务上达到与全注意力外推方法相当的表现，部分任务更优。
原文摘要 · Abstract (English)
Handling long-context sequences efficiently remains a significant challenge in large language models (LLMs). Existing methods for token selection in sequence extrapolation either employ a permanent eviction strategy or select tokens by chunk, which may lead to the loss of critical information. We propose Efficient Selective Attention (ESA), a novel approach that extends context length by efficiently selecting the most critical tokens at the token level to compute attention. ESA reduces the computational complexity of token selection by compressing query and key vectors into lower-dimensional representations. We evaluate ESA on long sequence benchmarks with maximum lengths up to 256k using open-source LLMs with context lengths of 8k and 32k. ESA outperforms other selective attention methods, especially in tasks requiring the retrieval of multiple pieces of information, achieving comparable performance to full-attention extrapolation methods across various tasks, with superior results in certain tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。