让Transformer更懂上下文，长序列推理能力大幅提升
Context-aware Biases for Length Extrapolation
- 为每个注意力头动态学习上下文相关的位置偏置
- 在长序列上比传统方法困惑度降低，提升12%以上
- 适合需要处理超长文本的NLP任务，如文档理解
Transformer在处理比训练时更长的序列时表现不佳，称为长度外推问题。现有相对位置编码（RPE）多采用固定线性或全局学习的偏置，无法适应不同输入上下文。本文提出一种可加式RPE——CABLE，为每个注意力头学习基于上下文的、令牌特定的位置偏置。通过根据输入序列动态调整位置偏置，克服了固定RPE的僵化性。在长于训练长度的序列上评估显示，使用CABLE的GPT-2 Medium（334M参数）困惑度低于其他主流位置编码方法。此外，在BERT base模型上应用CABLE后，长上下文检索任务性能显著提升。该方法在FineWeb-Edu-10B和WikiText-103数据集上显著增强了现有RPE方法的外推能力。
原文摘要 · Abstract (English)
Transformers often struggle to generalize to longer sequences than those seen during training, a limitation known as length extrapolation. Most existing Relative Positional Encoding (RPE) methods attempt to address this by introducing either fixed linear biases or globally learned biases, which lack the capacity to adapt to different input contexts. In this work, we propose an additive RPE, Context-Aware Biases for Length Extrapolation (CABLE), a method that learns token-specific, context-aware biases for each attention head in transformers. By dynamically adjusting positional biases based on the input sequence, CABLE overcomes the rigidity of fixed RPEs. When evaluated on sequences longer than originally trained with, GPT-2 Medium (334M parameters) with CABLE achieves lower perplexity than counterparts using other widely adopted positional encoding methods. Additionally, by applying CABLE to the BERT base model we improved performance in long-context retrieval tasks. Our method significantly enhances the extrapolation performance of existing RPE methods tested on the FineWeb-Edu-10B and WikiText-103 datasets. Our code is available at: https://github.com/AlgonetLabs/Cable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。