用分数阶微积分提升强化学习长期记忆，降低方差、提高采样效率。
Fractional Policy Gradients: Reinforcement Learning with Long-Term Memory
- 引入分数阶导数重构策略梯度，建立状态转移的幂律相关性。
- 理论证明方差随时间呈O(t^(-alpha))下降，实测样本效率提升35%-68%。
- 计算开销恒定，适合需长期依赖的复杂决策任务。
我们提出分数阶策略梯度（FPG），一种结合分数阶微积分的强化学习框架，用于策略优化中的长期时序建模。标准策略梯度方法受马尔可夫假设限制，存在高方差和采样效率低的问题。通过使用Caputo分数阶导数重构梯度，FPG在状态转移间建立了幂律时序相关性。我们开发了一种高效递归计算技术，实现分数阶时序差分误差的常数时间与内存开销。理论分析表明，相较于标准策略梯度，FPG在渐近意义下实现方差降低至O(t^(-alpha))，同时保持收敛性。实验验证显示，相比最先进基线，其样本效率提升35%-68%，方差降低24%-52%。该框架为利用长程依赖关系提供了数学严谨的方法，且无额外计算开销。
原文摘要 · Abstract (English)
We propose Fractional Policy Gradients (FPG), a reinforcement learning framework incorporating fractional calculus for long-term temporal modeling in policy optimization. Standard policy gradient approaches face limitations from Markovian assumptions, exhibiting high variance and inefficient sampling. By reformulating gradients using Caputo fractional derivatives, FPG establishes power-law temporal correlations between state transitions. We develop an efficient recursive computation technique for fractional temporal-difference errors with constant time and memory requirements. Theoretical analysis shows FPG achieves asymptotic variance reduction of order O(t^(-alpha)) versus standard policy gradients while preserving convergence. Empirical validation demonstrates 35-68% sample efficiency gains and 24-52% variance reduction versus state-of-the-art baselines. This framework provides a mathematically grounded approach for leveraging long-range dependencies without computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。