arXiv:2604.18972stat.MLcs.LG2026-04

提出高阶生成器回归,提升连续时间策略评估精度

Beyond Bellman: High-Order Generator Regression for Continuous-Time Policy Evaluation

论文配图:Beyond Bellman: High-Order Generator Regression for Continuous-Time Policy Evaluation
图 1 · 摘自论文原文
  • 用多步转移估计时变生成器,消除低阶截断误差
  • 二阶估计器在多场景下稳定优于贝尔曼基线
  • 理论明确给出高阶优势的适用条件和范围

我们研究在时变动态下,从离散闭环轨迹进行有限时域连续时间策略评估。目标价值曲面满足后向抛物方程,但基于单步递归的贝尔曼基线仅在网格宽度上为一阶。通过使用矩匹配系数估计多步转移的时变生成器,抵消低阶截断项,并结合反向回归。主要理论将总误差分解为生成器误设、投影误差、聚合偏差、有限样本误差和初始误差,并给出决策频率的区域图,解释何时可见高阶收益。在校准研究、四尺度基准测试、特征与初始值消融实验及增益不匹配压力测试中,二阶估计器始终优于贝尔曼基线,且在理论预测有收益的区间保持稳定。结果表明,高阶生成器回归是一种可解释的连续时间策略评估方法,具有明确的操作区域。

原文摘要 · Abstract (English)

We study finite-horizon continuous-time policy evaluation from discrete closed-loop trajectories under time-inhomogeneous dynamics. The target value surface solves a backward parabolic equation, but the Bellman baseline obtained from one-step recursion is only first-order in the grid width. We estimate the time-dependent generator from multi-step transitions using moment-matching coefficients that cancel lower-order truncation terms, and combine the resulting surrogate with backward regression. The main theory gives an end-to-end decomposition into generator misspecification, projection error, pooling bias, finite-sample error, and start-up error, together with a decision-frequency regime map explaining when higher-order gains should be visible. Across calibration studies, four-scale benchmarks, feature and start-up ablations, and gain-mismatch stress tests, the second-order estimator consistently improves on the Bellman baseline and remains stable in the regime where the theory predicts visible gains. These results position high-order generator regression as an interpretable continuous-time policy-evaluation method with a clear operating region.

策略评估连续时间生成器回归高阶方法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。