通过选择梯度空间中最具多样性任务，提升元强化学习样本效率。
Coreset-Based Task Selection for Sample-Efficient Meta-Reinforcement Learning
- 基于梯度空间多样性筛选任务子集，加权保留关键任务。
- 使找到ε近似最优解所需样本量减少O(1/ε)倍。
- 适合追求高效训练的元强化学习研究者使用。
我们研究如何通过任务选择提升模型无关元强化学习（MAML-RL）的样本效率。传统方法假设所有任务同等重要，但在任务高度相似时会导致冗余。为此，我们提出一种基于核心集（coreset）的任务选择方法，根据任务在梯度空间中的多样性进行加权筛选，优先保留最具信息量和差异性的任务。该方法可使达到ε-接近稳定解所需的样本数减少O(1/ε)倍，从而加速对未见任务的适应，并聚焦于最相关任务的训练。以MAML-LQR为例，当任务特定代价满足梯度主导性时，理论证明样本复杂度降低为O(log(1/ε))。数值实验在多个强化学习基准问题上验证了该方法的有效性，表明其优势不仅限于线性二次调节器（LQR）场景。
原文摘要 · Abstract (English)
We study task selection to enhance sample efficiency in model-agnostic meta-reinforcement learning (MAML-RL). Traditional meta-RL typically assumes that all available tasks are equally important, which can lead to task redundancy when they share significant similarities. To address this, we propose a coreset-based task selection approach that selects a weighted subset of tasks based on how diverse they are in gradient space, prioritizing the most informative and diverse tasks. Such task selection reduces the number of samples needed to find an $ε$-close stationary solution by a factor of O(1/$ε$). Consequently, it guarantees a faster adaptation to unseen tasks while focusing training on the most relevant tasks. As a case study, we incorporate task selection to MAML-LQR (Toso et al., 2024b), and prove a sample complexity reduction proportional to O(log(1/$ε$)) when the task specific cost also satisfy gradient dominance. Our theoretical guarantees underscore task selection as a key component for scalable and sample-efficient meta-RL. We numerically validate this trend across multiple RL benchmark problems, illustrating the benefits of task selection beyond the LQR baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。