无需训练即可提升大模型长文本处理能力，通过插值优化注意力分布。
A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI)
- 贪心重用预训练位置区间，插值注意力逻辑消除异常值
- 在多种长文本任务中表现稳定且优于现有方法
- 适合需要长上下文理解的场景，如文档摘要与推理
基于Transformer的大语言模型在输入超出训练上下文窗口时，因位置信息分布外（O.O.D.）问题导致注意力机制失准。现有方案包括微调和无训练方法，但存在效率低、冗余插值、逻辑值异常或局部位置信息丢失等问题。本文提出无训练的贪婪注意力逻辑插值方法（GALI），通过贪心复用预训练位置区间并插值注意力逻辑，消除异常值。GALI在广泛长文本任务中表现稳定且优越，无需针对输入长度进行调优。分析显示，大模型对位置区间的感知不均衡，限制插值范围可进一步提升性能，甚至改善短文本任务表现。GALI为大模型更鲁棒、通用的长文本处理迈出关键一步。代码与实验已开源于 https://github.com/adlnlp/Gali。
原文摘要 · Abstract (English)
Transformer-based Large Language Models (LLMs) struggle with inputs exceeding their training context window due to positional out-of-distribution (O.O.D.) issues that disrupt attention. Existing solutions, including fine-tuning and training-free methods, face challenges like inefficiency, redundant interpolation, logit outliers, or loss of local positional information. We propose Greedy Attention Logit Interpolation (GALI), a training-free method that improves length extrapolation by greedily reusing pretrained positional intervals and interpolating attention logit to eliminate outliers. GALI achieves stable and superior performance across a wide range of long-context tasks without requiring input-length-specific tuning. Our analysis further reveals that LLMs interpret positional intervals unevenly and that restricting interpolation to narrower ranges improves performance, even on short-context tasks. GALI represents a step toward more robust and generalizable long-text processing in LLMs. Our implementation of GALI, along with the experiments from our paper, is open-sourced at https://github.com/adlnlp/Gali.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。