提出统一的长序列位置编码方法，无需微调即可支持超长上下文。
MrRoPE: Mixed-radix Rotary Position Embedding
- 从进制转换视角统一多种位置编码扩展方法
- 无需微调，在128K上下文上召回率超85%
- 适合需要长文本处理的模型部署场景
旋转位置编码（RoPE）扩展旨在提升模型在预训练时未见的更长序列上的表现。然而现有方法多样且缺乏统一理论基础。本文提出MrRoPE（混合进制旋转位置编码），基于进制系统转换视角，将多种扩展策略统一为不同进制转换方案。在此理论基础上，设计两种无需训练的扩展方法：MrRoPE-Uni与MrRoPE-Pro，分别采用均匀与渐进式进制转换。两者均实现‘短训长用’泛化能力。无微调情况下，MrRoPE-Pro在128K上下文的针堆测试中保持超过85%召回率，并在Infinite-Bench检索与对话子集上准确率超过YaRN两倍。理论分析表明，MrRoPE-Pro有效提升了RoPE可达到编码长度的上限，验证了方法的可靠性与实用性。
原文摘要 · Abstract (English)
Rotary Position Embedding (RoPE)-extension refers to modifying or generalizing the Rotary Position Embedding scheme to handle longer sequences than those encountered during pre-training. However, current extension strategies are highly diverse and lack a unified theoretical foundation. In this paper, we propose MrRoPE (Mixed-radix RoPE), a generalized encoding formulation based on a radix system conversion perspective, which elegantly unifies various RoPE-extension approaches as distinct radix conversion strategies. Based on this theory, we introduce two training-free extensions, MrRoPE-Uni and MrRoPE-Pro, which leverage uniform and progressive radix conversion strategies, respectively, to achieve 'train short, test long' generalization. Without fine-tuning, MrRoPE-Pro sustains over 85% recall in the 128K-context Needle-in-a-Haystack test and achieves more than double YaRN's accuracy on Infinite-Bench retrieval and dialogue subsets. Theoretical analysis confirms that MrRoPE-Pro effectively raises the upper bound of RoPE's attainable encoding length, which further validates the reliability and utility of our theory and methodology.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。