提出新方法提升分布强化学习的精度与稳定性
Path-Coupled Bellman Flows for Distributional Reinforcement Learning
- 用耦合路径建模当前与下一时刻回报分布，避免边界错配
- 通过共享噪声和控制变量，降低更新方差，提升训练稳定性
- 适合追求高精度分布估计的强化学习研究者
分布强化学习（DRL）建模回报的完整分布，但现有有限支持或分位数方法依赖投影，而基于流的方法常在流起点出现边界错配，或因当前与后续噪声独立导致高方差自举。本文提出路径耦合贝尔曼流（PCBF），一种连续时间的DRL方法，通过流匹配学习回报分布。其核心是源一致的贝尔曼耦合路径：当前路径从 t=0 的基先验出发，在 t=1 达到贝尔曼目标，且在中间时间点与下一状态流保持路径级仿射关系（不要求所有 t 时刻边缘分布满足分布贝尔曼不动点）。PCBF 通过共享基础噪声耦合当前与下一状态回报流，并采用 λ 参数化控制变量目标：λ=0 恢复无偏样本贝尔曼目标，λ>0 则以可控偏差换取方差降低。在可解析处理的马尔可夫回报过程（MRPs）、OGBench 和 D4RL 上的实验表明，PCBF 在分布保真度和训练稳定性上均有提升，且在离线强化学习任务中表现具有竞争力。
原文摘要 · Abstract (English)
Distributional reinforcement learning (DRL) models the full return distribution, but existing finite-support or quantile-based methods rely on projections, while recent flow-based approaches can suffer from \emph{boundary mismatch} at the flow source or from \emph{high-variance} bootstrapping when current and successor noises are independent. We propose Path-Coupled Bellman Flows (PCBF), a continuous-time DRL method that learns return distributions with flow matching using \textbf{source-consistent Bellman-coupled paths}: the current path starts from the required base prior at $t{=}0$, reaches the Bellman target at $t{=}1$, and maintains a pathwise affine relation to the successor flow at intermediate times (without requiring time-$t$ marginals to satisfy a distributional Bellman fixed point for all $t$). PCBF couples current and successor return flows through shared base noise and uses a $λ$-parameterized control-variate target: $λ{=}0$ recovers an unbiased sample Bellman target, while $λ{>}0$ trades controlled bias for variance reduction. Experiments on analytically tractable MRPs, OGBench, and D4RL show improved distributional fidelity and training stability, and competitive offline RL performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。