通过插入分隔符提升大模型处理长数字序列的能力
SepSeq: A Training-Free Framework for Long Numerical Sequence Processing in LLMs
- 用分隔符作为注意力汇聚点，让模型聚焦局部段落
- 平均准确率提升35.6%，推理令牌消耗减少16.4%
- 无需训练，可直接接入9种主流大模型
尽管基于Transformer的大语言模型理论上支持超长上下文窗口，但在处理长数值序列时性能严重下降。我们将其归因于Softmax机制导致的注意力分散，使模型难以聚焦。为此，提出无需训练、即插即用的SepSeq框架，通过战略性插入分隔符缓解注意力分散。机制上，分隔符充当注意力汇点，重新校准注意力以聚焦局部片段，同时保留全局上下文。在9种广泛采用的大模型上进行的大量评估表明，SepSeq在多个领域平均相对准确率提升35.6%，同时推理总令牌消耗平均降低16.4%。
原文摘要 · Abstract (English)
While transformer-based Large Language Models (LLMs) theoretically support massive context windows, they suffer from severe performance degradation when processing long numerical sequences. We attribute this failure to the attention dispersion in the Softmax mechanism, which prevents the model from concentrating attention. To overcome this, we propose Separate Sequence (SepSeq), a training-free, plug-and-play framework to mitigate dispersion by strategically inserting separator tokens. Mechanistically, we demonstrate that separator tokens act as an attention sink, recalibrating attention to focus on local segments while preserving global context. Extensive evaluations on 9 widely-adopted LLMs confirm the effectiveness of our approach: SepSeq yields an average relative accuracy improvement of 35.6% across diverse domains while reducing total inference token consumption by 16.4% on average.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。