用重采样理论证明经验回放能降方差，提升数据少时的训练效率
Variance Reduction via Resampling and Experience Replay
- 将经验回放建模为重采样$U$-统计量，给出严格方差缩减证明
- 在LSTD和PDE算法中显著提升稳定性，数据稀缺时效果更明显
- 适用于策略评估、核岭回归等任务，计算成本可从$O(n^3)$降至$O(n^2)$
经验回放是强化学习中的基础技术，通过将过往经验存入回放缓冲区并重复使用来提升学习稳定性。尽管其实践效果显著，但理论性质仍不清晰。本文提出一个理论框架，将经验回放建模为重采样的$U$-和$V$-统计量，提供严格的方差缩减保证。我们将该框架应用于基于最小二乘时序差分（LSTD）和基于偏微分方程（PDE）的无模型算法进行策略评估，结果表明在数据稀缺场景下显著提升了稳定性和效率。此外，该框架还扩展至核岭回归，证明基于经验回放的方法可将传统$O(n^3)$的时间复杂度降低至$O(n^2)$，同时减少方差。大量数值实验验证了理论结论，展示了经验回放在多种机器学习任务中的广泛适用性与有效性。
原文摘要 · Abstract (English)
Experience replay is a foundational technique in reinforcement learning that enhances learning stability by storing past experiences in a replay buffer and reusing them during training. Despite its practical success, its theoretical properties remain underexplored. In this paper, we present a theoretical framework that models experience replay using resampled $U$- and $V$-statistics, providing rigorous variance reduction guarantees. We apply this framework to policy evaluation tasks using the Least-Squares Temporal Difference (LSTD) algorithm and a Partial Differential Equation (PDE)-based model-free algorithm, demonstrating significant improvements in stability and efficiency, particularly in data-scarce scenarios. Beyond policy evaluation, we extend the framework to kernel ridge regression, showing that the experience replay-based method reduces the computational cost from the traditional $O(n^3)$ in time to as low as $O(n^2)$ in time while simultaneously reducing variance. Extensive numerical experiments validate our theoretical findings, demonstrating the broad applicability and effectiveness of experience replay in diverse machine learning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。