用Transformer解耦回报尺度，让元强化学习同时搞定多个任务
AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformers
- 将智能体的策略和价值函数转为分类任务，摆脱回报尺度影响
- 在5个基准上实现显著多任务在线适应能力提升
- 适合做大规模无标签任务元学习的场景
在多样化数据集上训练的语言模型通过上下文学习实现泛化。强化学习策略可通过序列模型记忆中的元学习达成类似效果。然而，现有元强化学习研究主要聚焦于单一任务的小幅变化适应，难以扩展到更通用行为，且面临多任务优化挑战。少有方案兼容元学习从大规模无标签任务中学习的目标。为此，我们重新审视多任务强化学习瓶颈源于不同任务间回报尺度不均导致的训练损失不平衡。基于最近的基于Transformer（上下文）的元强化学习进展，提出一种简单而可扩展的解决方案：将智能体的策略与价值函数目标均转换为分类形式，使优化过程脱离当前回报尺度。在Meta-World ML45、Multi-Game Procgen、Multi-Task POPGym、Multi-Game Atari和BabyAI五个基准上的大规模对比实验表明，该设计显著提升了在线多任务适应能力和记忆性能，且无需显式任务标签。
原文摘要 · Abstract (English)
Language models trained on diverse datasets unlock generalization by in-context learning. Reinforcement Learning (RL) policies can achieve a similar effect by meta-learning within the memory of a sequence model. However, meta-RL research primarily focuses on adapting to minor variations of a single task. It is difficult to scale towards more general behavior without confronting challenges in multi-task optimization, and few solutions are compatible with meta-RL's goal of learning from large training sets of unlabeled tasks. To address this challenge, we revisit the idea that multi-task RL is bottlenecked by imbalanced training losses created by uneven return scales across different tasks. We build upon recent advancements in Transformer-based (in-context) meta-RL and evaluate a simple yet scalable solution where both an agent's actor and critic objectives are converted to classification terms that decouple optimization from the current scale of returns. Large-scale comparisons in Meta-World ML45, Multi-Game Procgen, Multi-Task POPGym, Multi-Game Atari, and BabyAI find that this design unlocks significant progress in online multi-task adaptation and memory problems without explicit task labels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。