用大模型+任务嵌入训练价值函数，实现高效多任务强化学习。
Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learners
- 采用高容量值函数与可学习任务嵌入，缓解多任务训练中的梯度冲突。
- 在280多个任务上表现优于现有方法，且能快速适应新任务。
- 适合需要高效多任务学习的机器人控制与视觉强化学习场景。
近期语言建模和视觉领域的进展源于在多样化多任务数据上训练大模型。这一范式在基于价值的强化学习（RL）中影响有限，因小模型单任务训练仍是主流。这主要是由于多任务RL中稀疏奖励与梯度冲突导致时序差分优化不稳定。实际工作流通常避免在线训练，转而通过克隆专家轨迹或蒸馏单任务策略集合来构建通用智能体。本文提出，使用通过交叉熵训练、以可学习任务嵌入为条件的高容量值模型，可有效解决任务干扰问题，实现稳健且可扩展的在线多任务训练。我们在7个多任务基准上测试，涵盖超过280个不同任务，包括高自由度人形控制和离散视觉强化学习。结果表明，尽管方法简单，该方案在单任务与多任务性能上均达当前最优，并具备优异的样本效率，可快速迁移至新任务。
原文摘要 · Abstract (English)
Recent advances in language modeling and vision stem from training large models on diverse, multi-task data. This paradigm has had limited impact in value-based reinforcement learning (RL), where improvements are often driven by small models trained in a single-task context. This is because in multi-task RL sparse rewards and gradient conflicts make optimization of temporal difference brittle. Practical workflows for generalist policies therefore avoid online training, instead cloning expert trajectories or distilling collections of single-task policies into one agent. In this work, we show that the use of high-capacity value models trained via cross-entropy and conditioned on learnable task embeddings addresses the problem of task interference in online RL, allowing for robust and scalable multi-task training. We test our approach on 7 multi-task benchmarks with over 280 unique tasks, spanning high degree-of-freedom humanoid control and discrete vision-based RL. We find that, despite its simplicity, the proposed approach leads to state-of-the-art single and multi-task performance, as well as sample-efficient transfer to new tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。