arXiv:2412.14834cs.LG2024-12中稿 · AAAI被引 2

通过最大化任务表示的熵,提升离线元强化学习的泛化能力。

Entropy Regularized Task Representation Learning for Offline Meta-Reinforcement Learning

  • 用熵正则化让任务表示摆脱行为策略影响
  • 在MuJoCo上对新任务的适应能力显著提升
  • 适合研究离线元学习与鲁棒表示学习的学者

离线元强化学习旨在通过在多种任务的离线数据上训练,使智能体具备快速适应新任务的能力。基于上下文的方法利用状态-动作-奖励的历史轨迹(即上下文)推断当前任务的表示,并以此条件化策略和价值函数。直观上,任务表示越能捕捉底层任务特征,智能体对新任务的泛化能力就越强。然而,基于上下文的方法面临分布不匹配问题:离线数据中的上下文与测试时的上下文不一致,导致任务表示过拟合于训练数据。为解决此问题,我们通过最大化任务表示条件下行为策略的熵,近似最小化任务表示分布与行为策略间的互信息。在MuJoCo环境上的实验表明,相比基线方法,我们的任务表示更忠实反映底层任务,在分布内和分布外任务上均表现更优。

原文摘要 · Abstract (English)

Offline meta-reinforcement learning aims to equip agents with the ability to rapidly adapt to new tasks by training on data from a set of different tasks. Context-based approaches utilize a history of state-action-reward transitions -- referred to as the context -- to infer representations of the current task, and then condition the agent, i.e., the policy and value function, on the task representations. Intuitively, the better the task representations capture the underlying tasks, the better the agent can generalize to new tasks. Unfortunately, context-based approaches suffer from distribution mismatch, as the context in the offline data does not match the context at test time, limiting their ability to generalize to the test tasks. This leads to the task representations overfitting to the offline training data. Intuitively, the task representations should be independent of the behavior policy used to collect the offline data. To address this issue, we approximately minimize the mutual information between the distribution over the task representations and behavior policy by maximizing the entropy of behavior policy conditioned on the task representations. We validate our approach in MuJoCo environments, showing that compared to baselines, our task representations more faithfully represent the underlying tasks, leading to outperforming prior methods in both in-distribution and out-of-distribution tasks.

元强化学习离线学习表示学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。