arXiv:2604.13517cs.LGcs.AI2026-04

动态路由会误导强化学习,导致模型依赖数值捷径而非真实时间抽象。

Representation over Routing: Diagnosing Temporal Routing Pathologies in Multi-Timescale PPO

论文配图:Representation over Routing: Diagnosing Temporal Routing Pathologies in Multi-Timescale PPO
图 1 · 摘自论文原文
  • 用软注意力动态选择不同时间尺度的值函数头
  • 发现短时头因目标易预测而被过度路由,实际控制效果差
  • 提出解耦目标:仅用长时优势更新策略,提升最差种子性能

强化学习中的时间信用分配常通过多折扣因子的值估计实现。自然延伸是让智能体动态路由这些时间头,使用可微注意力或基于不确定性的启发式权重。本文指出此类路由可能产生数值捷径,而非可靠的时间抽象。在LunarLander-v2上,我们构建受控的PPO实验环境,诊断失败模式。首先,提出“代理目标劫持”:可微软注意力路由器在接收到当前更新的梯度时,会直接倾向于数值有利的优势头,即使该路由不改善物理控制。由于不同折扣因子下的未归一化优势具有不同有效量级,导致尺度不匹配漏洞。其次,发现无梯度误差路由中的“时间不确定性悖论”:短时头因预测目标更易获得更大路由权重,尽管它们与延迟任务成功对齐度更低。作为结构应对,我们研究目标解耦:价值网络保留多时间尺度辅助头,但策略仅以长时优势进行更新。该方法并非普遍性能提升,但在本实验中消除可利用的策略侧路由路径,并提升最差种子的返回值。代码已公开于https://github.com/ben-dlwlrma/Representation-Over-Routing。

原文摘要 · Abstract (English)

Temporal credit assignment in reinforcement learning is often approached by introducing value estimates at multiple discount factors. A natural next step is to let the actor dynamically route among these temporal heads, using either differentiable attention or heuristic uncertainty weights. This paper argues that such routing can create a numerical shortcut rather than a reliable temporal abstraction. We study this issue in a controlled PPO setting on LunarLander-v2, using the environment as a visual sandbox for diagnosing failure modes. First, we formalize Surrogate Objective Hacking: a differentiable softmax router exposed to the PPO surrogate receives a direct gradient toward advantage heads that are numerically favorable for the current update, even when this routing change does not correspond to improved physical control. Because unnormalized advantages at different discount factors have different effective scales, this creates a scale-discrepancy vulnerability. Second, we identify the Paradox of Temporal Uncertainty in gradient-free error-based routing: short-horizon heads can receive the largest routing share because their prediction targets are easier, even when they are less aligned with delayed task success. As a structural response, we study Target Decoupling: the critic may retain multi-timescale auxiliary heads, but the actor is updated only with the long-horizon advantage. Target Decoupling is not presented as a broad performance booster; in this run set it removes the exploitable actor-side routing pathway and improves the observed worst-seed return. Code is available at https://github.com/ben-dlwlrma/Representation-Over-Routing.

强化学习时间抽象路由机制策略优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。