提出R2R2方法，解决强化学习中高重用率下的过拟合问题
R2R2: Robust Representation for Intensive Experience Reuse via Redundancy Reduction in Self-Predictive Learning
- 通过冗余减少设计非中心化目标，稳定自预测学习表征
- 在UTD=20时使TD7性能提升约22%，优于现有最优模型
- 适用于需要高效数据重用的机器人控制等数据稀缺场景
在真实世界机器人等数据稀缺的强化学习领域，高强度的数据重用虽能提升效率，却易引发过拟合。现有研究多关注评估器偏差，而对高更新-数据比（UTD）下自预测学习（SPL）中的表征不稳定性仍缺乏探索。为此，本文提出鲁棒表征方法R2R2，通过冗余减少实现正则化。理论分析表明，标准零中心化与SPL的谱特性冲突，因此设计了非中心化目标。R2R2在SPL原生算法TD7上验证有效；为进一步证明其通用性，还将SPL模块集成至当前最优的SimbaV2，构建SimbaV2-SPL。11个连续控制任务实验表明，当UTD=20时，R2R2使TD7性能提升约22%，并为SimbaV2-SPL带来额外增益，后者自身达到新最优水平。
原文摘要 · Abstract (English)
For reinforcement learning in data-scarce domains like real-world robotics, intensive data reuse enhances efficiency but induces overfitting. While prior works focus on critic bias, representation-level instability in Self-Predictive Learning (SPL) under high Update-to-Data (UTD) regimes remains underexplored. To bridge this gap, we propose Robust Representation via Redundancy Reduction (R2R2), a regularization method within SPL. We theoretically identify that standard zero-centering conflicts with SPL's spectral properties and design a non-centered objective accordingly. We verify R2R2 on SPL-native algorithms like TD7. Furthermore, to demonstrate its orthogonality to prior advancements, we extend the state-of-the-art SimbaV2, which originally lacks SPL, by integrating a tailored SPL module, termed SimbaV2-SPL. Experiments across 11 continuous control tasks confirm that R2R2 effectively mitigates overfitting; specifically, at a UTD ratio of 20, it improves TD7 by ~22% and provides additional gains on top of SimbaV2-SPL, which itself establishes a new state-of-the-art. The code can be found at: https://github.com/songsang7/R2R2
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。