arXiv:2605.12693cs.LG2026-05

解决延迟反馈下决策优化的梯度失效问题,提升模型决策精度。

IGT-OMD: Implicit Gradient Transport for Decision-Focused Learning under Delayed Feedback

  • 用隐式梯度传输修正延迟梯度,动态重算过时梯度。
  • 在延迟50轮时减少17%~55%的决策损失,理论证明误差线性化。
  • 适合在线学习中需实时调整策略的强化学习与优化任务。

决策聚焦学习在下游决策损失上端到端训练预测模型,但在线场景面临反馈延迟:结果可能在多次环境交互后才到达。我们识别出一种独特的失败模式——‘滞留放大’,即在延迟下,梯度滞后与内层求解器敏感性耦合,导致遗憾超出单层延迟理论预期。我们证明,任何黑箱延迟优化器均因内层求解器近似误差产生不可消除的遗憾成本,且梯度滞后会引入随延迟二次增长的传输误差,除非采用双层感知校正。提出算法IGT-OMD,将隐式梯度传输应用于在线镜像下降中的超梯度,利用存储的内层解在当前参数处重新评估过时梯度。该方法使传输误差从二次依赖降为线性依赖,并首次实现具有队列长度自适应步长的延迟双层优化的次线性遗憾界。受控实验揭示机制指纹:单位延迟下传输收益为0.0%(p=1.00),延迟50轮时单调增长至9.5%(p<0.001),明确分离校正效果。在线性二次调节器、《魔兽争霸》最短路径和Sinkhorn最优传输任务中,IGT-OMD相较单层基线降低17%~55%的决策损失,相位跃迁与理论预测一致。

原文摘要 · Abstract (English)

Decision-focused learning trains predictive models end-to-end against downstream decision loss, but online settings suffer delayed feedback: outcomes may not arrive for many environment interactions. We identify \emph{staleness amplification}, a failure mode unique to bilevel optimization under delay, in which gradient staleness couples with inner-solver sensitivity to inflate regret beyond single-level delay theory. We prove that any black-box delayed optimizer incurs an irreducible regret cost from inner-solver approximation error, and that gradient staleness contributes a quadratically growing transport error without bilevel-aware correction. Our algorithm, \textbf{IGT-OMD}, applies Implicit Gradient Transport to hypergradients within Online Mirror Descent, re-evaluating stale gradients at the current parameters using stored inner solutions. This method reduces transport error from a quadratic to a linear dependence on delay and achieves the first sublinear regret bound for delayed bilevel optimization with queue-length-adaptive step sizes. Controlled experiments provide a \emph{mechanistic fingerprint}: transport benefit is exactly $0.0\%$ ($p=1.00$) at unit delay and grows monotonically to $9.5\%$ at fifty rounds ($p<0.001$), isolating the correction's effect. On Linear Quadratic Regulator, Warcraft shortest-path, and Sinkhorn optimal transport, IGT-OMD reduces decision loss by $17$--$55\%$ relative to single-level baselines, with phase transitions matching the theory.

决策学习延迟优化双层优化在线学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。