通过软截断优化位置编码，让大模型在超长文本中表现更稳定。
CoPE: Clipped RoPE as A Scalable Free Lunch for Long Context LLMs
- 用软截断方式处理RoPE低频成分，统一解决分布外问题与语义建模。
- 在256k上下文长度下性能显著提升，超越现有方法。
- 无需复杂改造，适合追求长文本推理的模型部署者使用。
旋转位置编码（RoPE）是大语言模型实现长上下文能力的关键组件。现有方法主要基于两类思路：一是缓解分布外（OOD）问题，通过扩展RoPE频率以适应未见位置；二是语义建模，强调RoPE计算出的注意力分数应始终优先相似语义的词元。本文提出一种极简干预策略CoPE——对RoPE低频成分进行软截断。该方法同时消除分布外异常值、优化语义信号，并避免硬截断引发的频谱泄漏。大量实验表明，仅在原RoPE上应用此策略，即可在长达256k的上下文长度下实现显著性能提升，验证了理论分析，确立了新基准。代码、数据与模型已开源。
原文摘要 · Abstract (English)
Rotary Positional Embedding (RoPE) is a key component of context scaling in Large Language Models (LLMs). While various methods have been proposed to adapt RoPE to longer contexts, their guiding principles generally fall into two categories: (1) out-of-distribution (OOD) mitigation, which scales RoPE frequencies to accommodate unseen positions, and (2) Semantic Modeling, which posits that the attention scores computed with RoPE should always prioritize semantically similar tokens. In this work, we unify these seemingly distinct objectives through a minimalist intervention, namely CoPE: soft clipping lowfrequency components of RoPE. CoPE not only eliminates OOD outliers and refines semantic signals, but also prevents spectral leakage caused by hard clipping. Extensive experiments demonstrate that simply applying our soft clipping strategy to RoPE yields significant performance gains that scale up to 256k context length, validating our theoretical analysis and establishing CoPE as a new state-of-the-art for length generalization. Our code, data, and models are available at https://github.com/hrlics/CoPE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。