arXiv:2606.10646cs.LGcs.CL2026-06中稿 · ICML被引 2

通过追踪注意力图中的信息流,精准定位大模型推理中关键步骤。

How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMs

论文配图:How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMs
图 1 · 摘自论文原文
  • 构建注意力驱动的有向无环图,全局建模信息传播路径。
  • 识别出连接问题与答案的信息流主干,定位高影响力关键节点。
  • 适用于需要精准推理的强化学习任务,提升复杂推理表现。

在大语言模型的强化学习中,标记级信用分配仍是关键挑战,传统方法对所有标记同等对待,难以区分决定性推理步骤与常规格式或流畅填充。近期工作尝试利用模型内部信号进行细粒度信用分配,但多为局部启发式方法,忽略信息传播的全局结构。本文提出 FlowTracer,一个基于注意力诱导的有向无环图的RL框架,其中节点对应标记,边容量由聚合注意力权重决定,并从全局结构中推导标记信用。边容量被重加权以保留仅能到达答案区域的影响,同时强制局部流量守恒,确保中间标记不因路径长度或无关分支而失真或增益有效质量。在此图上,FlowTracer提取从问题到答案的信息流主干,按流量吞吐量评分标记,揭示高影响力枢纽和汇合检查点,这些节点调控长距离依赖。由此衍生的重要性用于构建标记级奖励,使学习信号精确聚焦于引导信息通往(或偏离)正确答案的关键标记,从而在多种推理任务中实现一致性能提升。

原文摘要 · Abstract (English)

Token-level credit assignment remains a key obstacle for reinforcement learning (RL) in large language models (LLMs), where RL recipes typically treat all tokens equally, failing to distinguish decisive reasoning steps from routine formatting or fluent filler. Recent attempts leverage model-internal signals to assign finer-grained credit, but these are often point-wise heuristics that ignore the global structure of information propagation. We propose FlowTracer, an RL framework that traces answer-targeted reasoning flow on an attention-induced directed acyclic graph in which nodes correspond to tokens and edge capacities come from aggregated attention weights and derives token credit from this global structure. The edge capacities are reweighted to retain only the influence that can reach the answer region, while enforcing local flow conservation so intermediate tokens neither lose nor gain effective mass due to path length or irrelevant branches. On this graph, FlowTracer extracts an information-flow backbone connecting the question to the answer and scores tokens by flow throughput, revealing high-impact hubs and aggregation checkpoints that mediate long-range dependencies. These derived importances are used to shape token-level rewards, enabling learning signals to focus precisely on the tokens that route information toward (or away from) correct answers and delivering consistent performance gains across a range of reasoning tasks.

强化学习信息流注意力机制推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。