用更少数据提升离线强化学习效果，还能省计算资源。
Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Dataset
- 将数据选择问题转化为可高效求解的优化任务
- 选出的少量数据使算法性能提升且计算量显著降低
- 适合关注数据效率与训练速度的研究者
离线强化学习允许智能体从预收集的数据集中学习而无需与环境交互。一个关键但未被充分探索的挑战是如何选取最优数据子集以提升算法性能与训练效率。减少数据规模还能揭示解决类似问题所需的最小数据量。为此，我们提出ReDOR（用于离线RL的缩减数据集），将数据集选择建模为梯度近似优化问题。我们证明，强化学习中广泛使用的演员-评论家框架可重构成子模优化目标，从而实现高效的子集选择。为此，我们改进了正交匹配追踪（OMP），并引入多个专为离线强化学习设计的新修改。实验表明，ReDOR识别出的数据子集不仅提升了算法性能，还大幅降低了计算复杂度。
原文摘要 · Abstract (English)
Offline reinforcement learning (RL) represents a significant shift in RL research, allowing agents to learn from pre-collected datasets without further interaction with the environment. A key, yet underexplored, challenge in offline RL is selecting an optimal subset of the offline dataset that enhances both algorithm performance and training efficiency. Reducing dataset size can also reveal the minimal data requirements necessary for solving similar problems. In response to this challenge, we introduce ReDOR (Reduced Datasets for Offline RL), a method that frames dataset selection as a gradient approximation optimization problem. We demonstrate that the widely used actor-critic framework in RL can be reformulated as a submodular optimization objective, enabling efficient subset selection. To achieve this, we adapt orthogonal matching pursuit (OMP), incorporating several novel modifications tailored for offline RL. Our experimental results show that the data subsets identified by ReDOR not only boost algorithm performance but also do so with significantly lower computational complexity.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。