arXiv:2601.07164cs.LG2026-01AAAI

解决离线元强化学习中特征过泛化问题,提升复杂任务适应性。

Offline Meta-Reinforcement Learning with Flow-Based Task Inference and Adaptive Correction of Feature Overgeneralization

  • 通过分解Q值的特征与权重,识别并修正特征过泛化问题。
  • 在多个环境中实现比基线更快的策略适应与元策略优化。
  • 适合研究离线强化学习与元学习融合的学者参考。

离线元强化学习(OMRL)结合了离线强化学习从多样化数据集学习的优势与元强化学习对新任务的适应能力,有望实现安全高效的智能体知识获取。然而,由于分布外(OOD)动作的存在,以及元强化学习设置中的任务分布广泛和马尔可夫决策过程(MDP)模糊性,OMRL仍面临外推误差问题。现有研究表明,Q网络的泛化性能影响离线强化学习中的外推误差。本文通过将Q值分解为特征与权重两部分,发现尽管高质量数据下该分解能提升适应性和收敛速度,但在复杂任务中常导致策略退化或崩溃。我们观察到,当特征遭遇分布外样本时,分解后的Q值会产生显著估计偏差,这一现象称为‘特征过泛化’。为此,我们提出FLORA,通过建模特征分布并估计不确定性来识别分布外样本,并引入回报反馈机制自适应调整特征组件。此外,为学习精确的任务表征,FLORA显式使用一系列可逆变换建模复杂的任务分布。理论与实证均表明,相较于基线方法,FLORA在多种环境中实现了快速适应与元策略改进。

原文摘要 · Abstract (English)

Offline meta-reinforcement learning (OMRL) combines the strengths of learning from diverse datasets in offline RL with the adaptability to new tasks of meta-RL, promising safe and efficient knowledge acquisition by RL agents. However, OMRL still suffers extrapolation errors due to out-of-distribution (OOD) actions, compromised by broad task distributions and Markov Decision Process (MDP) ambiguity in meta-RL setups. Existing research indicates that the generalization of the $Q$ network affects the extrapolation error in offline RL. This paper investigates this relationship by decomposing the $Q$ value into feature and weight components, observing that while decomposition enhances adaptability and convergence in the case of high-quality data, it often leads to policy degeneration or collapse in complex tasks. We observe that decomposed $Q$ values introduce a large estimation bias when the feature encounters OOD samples, a phenomenon we term ''feature overgeneralization''. To address this issue, we propose FLORA, which identifies OOD samples by modeling feature distributions and estimating their uncertainties. FLORA integrates a return feedback mechanism to adaptively adjust feature components. Furthermore, to learn precise task representations, FLORA explicitly models the complex task distribution using a chain of invertible transformations. We theoretically and empirically demonstrate that FLORA achieves rapid adaptation and meta-policy improvement compared to baselines across various environments.

强化学习元学习离线学习特征泛化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。