用旧数据提升强化学习效率，让训练更快更省资源。
Turning Sand to Gold: Recycling Data to Bridge On-Policy and Off-Policy Learning via Causal Bound
- 基于因果理论重新利用历史数据，改进策略评估。
- 实验显示奖励提升最高达383%，缓存大小减少96%。
- 适合追求高效训练的强化学习研究者与工程师。
深度强化学习（DRL）在复杂决策任务中表现优异，但通常需要大量训练步数和庞大的经验回放缓存，带来巨大的计算与资源开销。为解决此问题，我们引入一种新理论，将Neyman-Rubin潜在结果框架应用于DRL。不同于多数方法聚焦于反事实损失的界，我们建立了对实际损失（即DRL中的在线策略损失）的因果界，该界通过在经验回放缓存中存储过往价值网络输出来计算，有效利用了通常被丢弃的数据。在Atari 2600和MuJoCo多个任务上，对DQN和SAC等代理的广泛实验表明，引入该方法后，奖励比提升最高达383%，且缓存规模可降低96%，显著提升样本效率，代价几乎可忽略。
原文摘要 · Abstract (English)
Deep reinforcement learning (DRL) agents excel in solving complex decision-making tasks across various domains. However, they often require a substantial number of training steps and a vast experience replay buffer, leading to significant computational and resource demands. To address these challenges, we introduce a novel theoretical result that leverages the Neyman-Rubin potential outcomes framework into DRL. Unlike most methods that focus on bounding the counterfactual loss, we establish a causal bound on the factual loss, which is analogous to the on-policy loss in DRL. This bound is computed by storing past value network outputs in the experience replay buffer, effectively utilizing data that is usually discarded. Extensive experiments across the Atari 2600 and MuJoCo domains on various agents, such as DQN and SAC, achieve up to 383% higher reward ratio, outperforming the same agents without our proposed term, and reducing the experience replay buffer size by up to 96%, significantly improving sample efficiency at a negligible cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。