通过两阶段蒸馏让大模型同时精通多个任务，效果超越单一训练方式。
Building Multi-Task Agentic LLMs via Two-Phase Distillation
- 先用离策略蒸馏整合多任务专家，再用在线策略微调提升性能。
- 两阶段方法在对话与文本游戏任务中达到单任务专家水平。
- 适合需要多任务通用能力的智能体开发人员参考。
迈向通用人工智能的关键一步是训练能执行多项任务的模型。本文研究通过先为每个任务单独训练强化学习专家，再通过蒸馏整合的方式构建多任务模型，作为直接在混合任务上训练单一模型的替代方案。我们发现，在多任务设置下，离策略蒸馏因前向KL的模式覆盖特性而表现下降：来自多个任务的数据引入大量行为模式，超出学生模型容量，迫使模型平均行为，导致性能退化。相反，在线策略蒸馏具有模式搜索能力，但依赖强初始化。受此启发，我们提出两阶段方法：先进行离策略蒸馏，再进行在线策略精炼。在对话智能体和文本游戏任务上的评估表明，该方法在每个任务上均达到单任务强化学习专家的性能水平，而单独使用离策略或在线策略蒸馏均无法实现此效果。
原文摘要 · Abstract (English)
A key step toward artificial general intelligence is to train models that can perform multiple tasks. In this paper, we study how to build such models by first training separate RL experts for individual tasks and then consolidating them via distillation, as an alternative to directly training a single model on mixed tasks. We show that off-policy distillation degrades in multi-task settings due to the mode-covering nature of forward KL: aggregating data from multiple tasks introduces a large number of behavioral modes that can exceed the student's capacity, forcing it to average across behaviors and leading to degraded performance. In contrast, on-policy distillation is mode-seeking but requires strong initialization. Inspired by these observations, we propose a two-phase approach: off-policy distillation followed by on-policy refinement. Evaluation across conversational agents and text-based games confirms that this two-phase approach matches single-task RL expert performance for each individual task, whereas off-policy or on-policy distillation alone fails to match this performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。