用训练数据集的多样化数据蒸馏策略集合，提升强化学习泛化能力。
How Ensembles of Distilled Policies Improve Generalisation in Reinforcement Learning
- 训练后通过多策略蒸馏增强泛化性能
- 在多样训练数据上蒸馏的策略集表现更优
- 适合追求鲁棒性的强化学习应用
在零样本策略迁移的强化学习设置中，目标是让智能体在一组固定训练环境上训练后，能泛化到未见过的相似测试环境中。以往研究发现，训练后进行策略蒸馏有时可使新策略在测试环境中表现优于原始策略。但其原因尚不明确，且蒸馏所用数据的选择也不清晰。本文在一定假设下，推导出训练后策略蒸馏的泛化界，理论揭示两个实用洞见：为提升泛化性，应训练一个蒸馏策略的集合，并尽可能使用来自训练环境的大量数据进行蒸馏。我们在更一般场景下实证验证了这些洞见的有效性，即使理论所需假设不再成立。最终证明，基于多样化数据蒸馏的策略集合，在泛化性能上显著优于原始智能体。
原文摘要 · Abstract (English)
In the zero-shot policy transfer setting in reinforcement learning, the goal is to train an agent on a fixed set of training environments so that it can generalise to similar, but unseen, testing environments. Previous work has shown that policy distillation after training can sometimes produce a policy that outperforms the original in the testing environments. However, it is not yet entirely clear why that is, or what data should be used to distil the policy. In this paper, we prove, under certain assumptions, a generalisation bound for policy distillation after training. The theory provides two practical insights: for improved generalisation, you should 1) train an ensemble of distilled policies, and 2) distil it on as much data from the training environments as possible. We empirically verify that these insights hold in more general settings, when the assumptions required for the theory no longer hold. Finally, we demonstrate that an ensemble of policies distilled on a diverse dataset can generalise significantly better than the original agent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。