arXiv:2605.14350cs.LG2026-05

通过动态调整任务采样,让智能体更高效攻克难任务。

Distributionally Robust Multi-Task Reinforcement Learning via Adaptive Task Sampling

论文配图:Distributionally Robust Multi-Task Reinforcement Learning via Adaptive Task Sampling
图 1 · 摘自论文原文
  • 根据任务求解进度动态分配数据,优先选最难的任务
  • 在MetaWorld-MT10/MT50上提升最差任务表现23%以上
  • 适合多任务训练中存在任务难度差异的场景

多任务强化学习(MTRL)旨在训练单一智能体同时优化多个任务的表现。然而,联合优化常导致学习失衡:智能体快速解决简单任务,却在复杂任务上进展缓慢。现有方法多归因于冲突的任务梯度,通过梯度调节或专用架构缓解。本文另辟蹊径,关注被忽视的数据分配不均问题——标准MTRL对所有任务分配等量环境交互,导致简单任务数据冗余、困难任务数据不足。为此,提出分布鲁棒自适应任务采样(DRATS),通过将MTRL形式化为可行性问题,构建最小化最坏情况回报差距(目标回报与实际回报之差)的极小极大目标,实现对未完成任务的动态优先采样。在MetaWorld-MT10和MT50基准测试中,相比现有采样算法,DRATS显著提升数据效率,并使最差任务性能平均提升23%以上。

原文摘要 · Abstract (English)

Multi-task reinforcement learning (MTRL) aims to train a single agent to efficiently optimize performance across multiple tasks simultaneously. However, jointly optimizing all tasks often yields imbalanced learning: agents quickly solve easy tasks but learn slowly on harder ones. While prior work primarily attributes this imbalance to conflicting task gradients and proposes gradient manipulation or specialized architectures to address it, we instead focus on a distinct and under-explored challenge: imbalanced data allocation. Standard MTRL allocates an equal number of environment interactions to each task, which over-allocates data to easy tasks that require relatively few interactions to solve and under-allocates data to hard tasks that require substantially more experience to solve. To address this challenge, we introduce Distributionally Robust Adaptive Task Sampling (DRATS), an algorithm that adaptively prioritizes sampling tasks furthest from being solved. We derive DRATS by formalizing MTRL as a feasibility problem from which we derive a minimax objective for minimizing the worst-case return gap, the difference between a desired target return and the agent's return on a task. In benchmarks like MetaWorld-MT10 and MT50, DRATS improves data efficiency and increases worst-task performance compared to existing task sampling algorithms.

多任务强化学习自适应采样数据效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。