将强化学习环境压缩成单批数据，实现一步监督学习。
Distilling Reinforcement Learning into Single-Batch Datasets
- 用元学习扩展PPO，将RL环境转化为单批次监督数据。
- 在多种MuJoCo和Atari环境中实现一步学习，性能接近原任务。
- 适用于希望快速训练的开发者，尤其适合资源受限场景。
数据蒸馏将大规模数据集压缩为小规模合成数据集，使在合成数据上学习能近似原始数据的学习效果。在合成数据上训练仅需一次梯度下降即可完成。本文证明了该方法可泛化至不同任务，将强化学习环境蒸馏为单批监督学习数据集。这不仅展示了对强化学习任务的压缩能力，还实现了从强化学习到监督学习的模态转换。我们提出一种用于元学习的改进版近端策略优化算法，并应用于多维经典Cart-Pole问题、所有MuJoCo环境及多个Atari游戏的蒸馏。结果表明，复杂强化学习环境可被压缩为单步监督学习任务，验证了蒸馏在不同学习架构间的泛化能力，并实现了最小可能的合成数据集。
原文摘要 · Abstract (English)
Dataset distillation compresses a large dataset into a small synthetic dataset such that learning on the synthetic dataset approximates learning on the original. Training on the distilled dataset can be performed in as little as one step of gradient descent. We demonstrate that distillation is generalizable to different tasks by distilling reinforcement learning environments into one-batch supervised learning datasets. This demonstrates not only distillation's ability to compress a reinforcement learning task but also its ability to transform one learning modality (reinforcement learning) into another (supervised learning). We present a novel extension of proximal policy optimization for meta-learning and use it in distillation of a multi-dimensional extension of the classic cart-pole problem, all MuJoCo environments, and several Atari games. We demonstrate distillation's ability to compress complex RL environments into one-step supervised learning, explore RL distillation's generalizability across learner architectures, and demonstrate distilling an environment into the smallest-possible synthetic dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。