arXiv:2602.14587cs.LGcs.AI2026-02

提出解耦的连续时间强化学习,解决动作排序失效问题。

Decoupled Continuous-Time Reinforcement Learning via Hamiltonian Flow

  • 用哈密顿流解耦价值函数与优势率函数,交替更新
  • 在真实交易任务中实现21%季度收益,接近翻倍最佳表现
  • 理论保证收敛,不依赖敏感测试过程,适合复杂控制场景

许多现实世界控制问题(如金融、机器人)以连续时间演化,决策非均匀且事件驱动。标准离散时间强化学习基于固定步长贝尔曼更新,在此场景下失效:当时间间隔缩小,Q函数坍缩为价值函数V,丧失动作排序能力。现有连续时间方法通过优势率函数q恢复动作信息,但依赖复杂的鞅损失或正交性约束,对测试过程选择敏感。这些方法将V与q纠缠成庞大复杂优化问题,难以稳定训练。为此,我们提出一种新型解耦连续时间演员-评论家算法,采用交替更新机制:q由V上的扩散生成器学习,V则通过哈密顿值流更新,在无穷小时间步下仍保持信息量,而传统max/softmax备份在此失效。理论上,我们通过新的概率论证证明了严格收敛,绕开了生成器基哈密顿量在上范数下缺乏贝尔曼收缩的难题。实验上,该方法在多个挑战性连续控制基准和一个真实世界交易任务中优于先前连续时间及主流离散时间基线,季度利润达21%,近乎翻倍第二名方法。

原文摘要 · Abstract (English)

Many real-world control problems, ranging from finance to robotics, evolve in continuous time with non-uniform, event-driven decisions. Standard discrete-time reinforcement learning (RL), based on fixed-step Bellman updates, struggles in this setting: as time gaps shrink, the $Q$-function collapses to the value function $V$, eliminating action ranking. Existing continuous-time methods reintroduce action information via an advantage-rate function $q$. However, they enforce optimality through complicated martingale losses or orthogonality constraints, which are sensitive to the choice of test processes. These approaches entangle $V$ and $q$ into a large, complex optimization problem that is difficult to train reliably. To address these limitations, we propose a novel decoupled continuous-time actor-critic algorithm with alternating updates: $q$ is learned from diffusion generators on $V$, and $V$ is updated via a Hamiltonian-based value flow that remains informative under infinitesimal time steps, where standard max/softmax backups fail. Theoretically, we prove rigorous convergence via new probabilistic arguments, sidestepping the challenge that generator-based Hamiltonians lack Bellman-style contraction under the sup-norm. Empirically, our method outperforms prior continuous-time and leading discrete-time baselines across challenging continuous-control benchmarks and a real-world trading task, achieving 21% profit over a single quarter$-$nearly doubling the second-best method.

强化学习连续时间哈密顿流解耦

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。