提出二阶平滑规划，用最优传输优化贝尔曼更新，减少模拟调用次数。
Second-Order Smooth Planning with Optimal-Transport Bellman Smoothing
- 基于最优传输平滑贝尔曼更新，实现二阶误差控制。
- 样本复杂度降至 $ ilde{O}(\varepsilon^{-3})$，优于原有 $ ilde{O}(\varepsilon^{-4})$。
- 适合高效强化学习规划，尤其在低样本场景下表现更优。
基于生成模型的规划旨在尽可能少地调用模拟器来估计状态价值。SmoothCruiser通过利用熵正则化贝尔曼更新的光滑性，实现了问题无关的样本复杂度 $ ilde{O}(\varepsilon^{-4})$，但其估计器仅为一阶。我们证明，SmoothCruiser类规划器的样本复杂度指数由局部泰勒余项阶数 $eta$ 决定，理论最优复杂度为 $ ilde{O}( ilde{ au}^{-(2+2/(eta-1))})$：当 $eta=2$（一阶）时恢复 SmoothCruiser，而 $eta=3$（二阶/三次余项）可达到 $ ilde{O}( ilde{ au}^{-3})$。我们通过在动作分布上采用最优传输平滑的贝尔曼更新，实现该二阶精度，该方法具有闭式解、策略梯度和利普希茨赫斯蒂安，其二次修正可通过无偏交叉乘积估计器计算。由此得到的 SecondOrderSmoothCruiser 在固定最优传输参数下达到 $ ilde{O}( ilde{ au}^{-3})$ 的预言机复杂度，并通过显式的正则化-偏差界建立了最优传输、熵正则化与无正则化目标之间的联系。
原文摘要 · Abstract (English)
Planning with a generative model aims to estimate the value of a state using as few simulator calls as possible. SmoothCruiser achieves problem-independent complexity $\widetilde O(\varepsilon^{-4})$ by exploiting the smoothness of the entropy-regularized Bellman backup, but its estimator is only first-order. We show that the sample-complexity exponent of SmoothCruiser-type planners is governed by the order $\beta$ of the local Taylor remainder, giving oracle complexity $\widetilde O(\varepsilon^{-(2+2/(\beta-1))})$: the first-order case $\beta=2$ recovers SmoothCruiser, while a second-order/cubic remainder $\beta=3$ yields $\widetilde O(\varepsilon^{-3})$. We reach this regime with an optimal-transport-smoothed Bellman backup over action distributions, which has a closed form, a policy gradient, and a Lipschitz Hessian, and whose quadratic correction admits an unbiased cross-product estimator. The resulting SecondOrderSmoothCruiser achieves $\widetilde O(\varepsilon^{-3})$ oracle complexity for fixed OT parameters, and we relate the OT, entropy-regularized, and unregularized objectives through explicit regularization-bias bounds.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。