arXiv:2510.13554cs.CLcs.LG2025-10中稿 · ICML被引 26

用注意力机制揭示大模型推理节奏,实现精准策略优化

Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization

  • 将注意力头分为局部与全局聚焦,发现预规划与锚点的周期性模式
  • 提出两个量化指标,识别关键推理节点并动态分配强化学习信用
  • 适用于希望提升大模型推理透明度与效率的研究者和开发者

大语言模型(LLM)的推理过程仍不清晰,传统强化学习对整个生成过程统一赋权,模糊了关键步骤与常规步骤的差异。本文将注意力视为可解释推理的机制蓝图,而非计算副产品。通过区分局部与全局关注的注意力头,发现局部头在对角线附近呈现锯齿状模式,对应短语块;全局头则揭示对后续词元有广泛影响的关键词元。为此提出两个量化指标:1)窗口平均注意力距离,衡量在限定窗口内的回溯注意力范围;2)未来注意力影响,量化一个词元被后续词元关注的平均程度。结合两者揭示出反复出现的‘预规划-锚点’机制:模型先进行长程上下文参考生成引导词元,随即或同时生成组织后续推理的语义锚点。基于此,我们设计三种新型强化学习策略,针对关键节点(预规划词元、锚点词元及其时间耦合)实施动态信用分配,在多个推理任务中均取得稳定性能提升。通过与模型内在推理节奏对齐,将模糊的优化转化为结构感知的可操作过程,为提升大模型推理的透明性与有效性提供可能。

原文摘要 · Abstract (English)

The reasoning pattern of Large language models (LLMs) remains opaque, and reinforcement learning (RL) typically applies uniform credit across an entire generation, blurring the distinction between pivotal and routine steps. This work positions attention as a privileged substrate that renders the internal logic of LLMs legible, not merely as a byproduct of computation, but as a mechanistic blueprint of reasoning itself. We first distinguish attention heads between locally and globally focused information processing and reveal that locally focused heads produce a sawtooth pattern near the diagonal indicating phrasal chunks, while globally focused heads expose tokens that exert broad downstream influence over future tokens. We formalize these with two metrics: 1) Windowed Average Attention Distance, which measures the extent of backward attention within a clipped window; 2) Future Attention Influence, which quantifies a token's global importance as the average attention it receives from subsequent tokens. Taken together, these signals reveal a recurring preplan-and-anchor mechanism, where the model first performs a long-range contextual reference to generate an introductory token, which is immediately followed by or coincides with a semantic anchor token that organizes subsequent reasoning. Leveraging these insights, we introduce three novel RL strategies that dynamically perform targeted credit assignment to critical nodes (preplan tokens, anchor tokens, and their temporal coupling) and show consistent performance gains across various reasoning tasks. By aligning optimization with the model's intrinsic reasoning rhythm, we aim to transform opaque optimization into an actionable structure-aware process, hoping to offer a potential step toward more transparent and effective optimization of LLM reasoning.

大模型推理注意力机制强化学习可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。