通过技能发现实现高效多任务离线强化学习，无需重新训练新任务。
Beyond Shallow Behavior: Task-Efficient Value-Based Multi-Task Offline MARL via Skill Discovery
- 在隐空间中重构下一时刻观测来发现技能，避免行为克隆依赖。
- 在《星际争霸2》上14个任务集中有13个表现最佳，单任务提升达68.9%。
- 适合需要快速适应新任务的高成本、高风险场景使用。
作为数据驱动方法，离线多智能体强化学习（Offline MARL）仅从离线数据集中学习优质策略，适用于历史数据丰富但交互成本高、风险大的领域。然而，现有方法大多针对特定任务，需为新任务重新训练，造成冗余与低效。为此，本文提出一种任务高效的基于价值的多任务离线MARL算法——技能发现保守Q学习（SD-CQL）。不同于传统方法通过行为克隆从技能解码动作，SD-CQL在隐空间中通过重构下一时刻观测来发现技能，分别评估固定与可变动作，并采用带有局部价值校准的保守Q学习，为每个技能选择最优动作。该方法消除局部-全局对齐需求，实现从有限的小规模源任务中强泛化。在《星际争霸2》上的大量实验表明，SD-CQL在14个任务集中的13个上表现最优，单任务集最高提升达68.9%。
原文摘要 · Abstract (English)
As a data-driven approach, offline MARL learns superior policies solely from offline datasets, ideal for domains rich in historical data but with high interaction costs and risks. However, most existing methods are task-specific, requiring retraining for new tasks, leading to redundancy and inefficiency. To address this issue, we propose a task-efficient value-based multi-task offline MARL algorithm, Skill-Discovery Conservative Q-Learning (SD-CQL). Unlike existing methods decoding actions from skills via behavior cloning, SD-CQL discovers skills in a latent space by reconstructing the next observation, evaluates fixed and variable actions separately, and uses conservative Q-learning with local value calibration to select the optimal action for each skill. It eliminates the need for local-global alignment and enables strong multi-task generalization from limited, small-scale source tasks. Substantial experiments on StarCraft II demonstrate the superior generalization performance and task-efficiency of SD-CQL. It achieves the best performance on $\textbf{13}$ out of $14$ task sets, with up to $\textbf{68.9%}$ improvement on individual task sets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。