arXiv:2604.27083cs.LG2026-04被引 4

让多个专家模型边训练边互相教,实现多能力融合。

Co-Evolving Policy Distillation

论文配图:Co-Evolving Policy Distillation
图 1 · 摘自论文原文
  • 专家并行训练时实时互为教师,双向蒸馏提升一致性。
  • 在文本、图像、视频推理上超越混合强化蒸馏与单一专家。
  • 适合需要多模态综合能力的AI系统研发者使用。

RLVR和OPD已成为后训练的标准范式。本文统一分析这两种范式将多个专家能力整合到单个模型中的机制,发现混合式RLVR存在跨能力分歧代价,而先训专家再进行OPD虽避免分歧,却因师生行为模式差异大,未能充分吸收教师知识。为此提出协同进化策略蒸馏(CoPD):在每个专家的RLVR训练过程中嵌入OPD,专家之间相互作为教师,实现双向蒸馏与协同进化。该方法既保持了专家间行为的一致性,又保留足够的互补知识。实验表明,CoPD成功实现文本、图像、视频推理能力的一体化整合,在性能上显著优于混合式RLVR和MOPD等强基线,甚至超过领域专用专家。其模型并行训练模式可能启发新的训练扩展范式。

原文摘要 · Abstract (English)

RLVR and OPD have become standard paradigms for post-training. We provide a unified analysis of these two paradigms in consolidating multiple expert capabilities into a single model, identifying capability loss in different ways: mixed RLVR suffers from inter-capability divergence cost, while the pipeline of first training experts and then performing OPD, though avoiding divergence, fails to fully absorb teacher capabilities due to large behavioral pattern gaps between teacher and student. We propose Co-Evolving Policy Distillation (CoPD), which encourages parallel training of experts and introduces OPD during each expert's ongoing RLVR training rather than after complete expert training, with experts serving as mutual teachers (making OPD bidirectional) to co-evolve. This enables more consistent behavioral patterns among experts while maintaining sufficient complementary knowledge throughout. Experiments validate that CoPD achieves all-in-one integration of text, image, and video reasoning capabilities, significantly outperforming strong baselines such as mixed RLVR and MOPD, and even surpassing domain-specific experts. The model parallel training pattern offered by CoPD may inspire a novel training scaling paradigm.

强化学习知识蒸馏多模态模型融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。