提出双焦点注意力机制，提升模型对递归逻辑的泛化能力
Bifocal Attention: Harmonizing Geometric and Spectral Positional Embeddings for Algorithmic Generalization
- 分离几何与频谱位置编码，分别处理局部和长程结构
- 通过频谱演化让位置频率自适应优化，适配算法拓扑
- 解决传统旋转位置编码在深层递归推理中的结构缺失问题
旋转位置编码（RoPE）因其通过几何旋转编码相对位置，已成为大语言模型的标准。然而我们识别出一个关键局限，称为“频谱刚性”：标准RoPE采用固定的几何衰减（θ⁻ᵢ），虽优化于局部句法一致性，却无法捕捉递归逻辑和算法推理中固有的长程周期结构，导致“结构差距”，使在浅层推理链上训练的模型难以外推至更深的递归步骤。本文提出双焦点注意力（Bifocal Attention），将位置编码解耦为两种模态：几何眼（标准RoPE）用于精确的词元级操作，频谱眼（可学习谐波算子）用于追踪长程递归深度。我们设计了一种新训练协议——频谱演化，初始将位置频率设为静态几何参数，但允许其通过梯度下降演变为针对特定算法拓扑优化的谐波基。
原文摘要 · Abstract (English)
Rotary Positional Embeddings (RoPE) have become the standard for Large Language Models (LLMs) due to their ability to encode relative positions through geometric rotation. However, we identify a significant limitation we term ''Spectral Rigidity'': standard RoPE utilizes a fixed geometric decay ($θ^{-i}$) optimized for local syntactic coherence, which fails to capture the long-range, periodic structures inherent in recursive logic and algorithmic reasoning. This results in a ''Structure Gap'', where models trained on shallow reasoning chains fail to extrapolate to deeper recursive steps. In this work, we introduce Bifocal Attention, an architectural paradigm that decouples positional encoding into two distinct modalities: Geometric Eyes (Standard RoPE) for precise token-level manipulation, and Spectral Eyes (Learnable Harmonic Operators) for tracking long-range recursive depth. We propose a novel training protocol, Spectral Evolution, which initializes positional frequencies as static geometric parameters but allows them to evolve via gradient descent into a harmonic basis optimized for the specific algorithmic topology of the task.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。