arXiv:2605.27115cs.AI2026-05被引 6

提出新方法在不依赖教师数据分布的情况下,同时恢复大模型通用能力并保留领域专长。

Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation

论文配图:Counteraction-Aware Multi-Teacher On-Policy Distillation for General Capability Recovery with Domain Preservation
图 1 · 摘自论文原文
  • 分阶段交替训练,分离通用能力恢复与领域特性保持
  • 通过选择差异大的样本集中修正信号,提升优化效率
  • 适合需要兼顾通用性与领域性能的模型微调场景

领域专化可提升大模型在垂直领域的表现,但常削弱其原始模型的通用能力。现有多教师在线蒸馏(MOPD)方法依赖教师提示覆盖均匀,需提示匹配教师训练分布,但在通用教师为开源模型、后训练数据未知时难以满足。本文不尝试重构隐藏分布,而是使用现成的代理通用提示进行通用能力恢复。发现原始MOPD在覆盖不全时存在两类失效:恢复与保持间的梯度冲突,以及弱信号平均导致的修正乏力。为此提出反作用感知的多教师在线蒸馏(CaMOPD),通过解耦交替训练和基于差距的样本选择解决上述问题。该方法专门分配更新以恢复通用能力,周期性审查领域提示以维持专长,并优先选择教师-学生概率差距较大的样本以集中修正信号。在角色扮演对话与医学问答任务中,CaMOPD在通用能力恢复上优于基线,同时保持领域行为。梯度一致性分析进一步验证了其生成更连贯修正信号的有效性。

原文摘要 · Abstract (English)

Domain specialization can improve LLM behavior in vertical domains, but often weakens the general capabilities inherited from the original model. Recent Multi-Teacher On-Policy Distillation (MOPD) pipelines recover model capabilities by supervising student-generated trajectories with teacher feedback, but typically assume teacher-aligned prompt coverage, requiring prompts to match the teachers' training distributions. This assumption is difficult to satisfy when the general teacher is an open-source model whose post-training data are unknown. Instead of attempting to reconstruct this hidden distribution, we study general capability recovery with readily available proxy general prompts. We identify two failure modes of vanilla MOPD in this incomplete-coverage situation: recovery-preservation counteraction from mixing conflicting recovery and preservation gradients, and weak-signal flattening from uniformly averaging samples with unequal correction demand. We propose Counteraction-Aware Multi-Teacher On-Policy Distillation (CaMOPD), which addresses these issues with decoupled alternating training and gap-based sample selection. CaMOPD gives general recovery dedicated updates, periodically reviews domain prompts for preservation, and selects samples with larger averaged token-level teacher-student log-probability gaps to concentrate correction signals. Across role-play dialogue and medical reasoning QA scenarios, CaMOPD performs best in general recovery over baselines while maintaining domain-specific behavior. Gradient coherence analyses further support the intended effect of CaMOPD in producing more coherent correction signals.

大模型微调知识保留蒸馏方法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。