arXiv:2606.00147cs.LGcs.AI2026-06被引 1

通过数据重构与轨迹保真,让模型在领域微调中既提升性能又减少遗忘。

RAFT: Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting

论文配图:RAFT: Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting
图 1 · 摘自论文原文
  • 用自条件重写和答案融合生成更匹配模型风格的训练数据。
  • 在学生生成的路径上用原模型提供软目标,稳定保留通用能力。
  • 适合需要兼顾领域性能与通用能力的模型微调场景。

领域特定监督微调常以牺牲模型通用能力为代价提升领域表现。本文指出两个关键问题:一是监督兼容性差距,领域目标在风格和推理格式上与原始模型自然输出不一致;二是轨迹保真差距,教师强迫微调仅优化固定目标词元,未约束模型在自生成前缀上的行为,导致原始行为丢失。为此提出RAFT框架:第一阶段通过自条件重写、语义过滤和答案融合构建模型兼容的监督信号;第二阶段采用答案条件的在线策略蒸馏,使原指令微调模型基于融合答案作为上下文,对学生的生成路径提供软目标。引入top-K温度蒸馏和EMA自适应损失平衡以稳定领域-通用权衡。在三个指令微调骨干模型和五个领域上,RAFT相较标准SFT平均提升领域准确率23.2%,同时在MS-Bench和IFEval上分别恢复18.2%和10.2%的性能退化,证明结合数据重构与轨迹级保真是缓解遗忘的有效方案。

原文摘要 · Abstract (English)

Domain-specific supervised fine-tuning (SFT) often improves in-domain performance at the cost of degrading a model's general capabilities. We view this degradation through two practical gaps in domain SFT: a supervision-compatibility gap, where domain targets differ in style and reasoning format from the original model's natural responses, and a trajectory-preservation gap, where teacher-forced SFT optimizes fixed target tokens without constraining the model's behavior on its own generated prefixes. This process fails to preserve the model's original behavior. We propose RAFT (Data Refinement and Adaptive Distillation for Domain Fine-Tuning with Alleviated Forgetting), a two-stage framework that addresses both factors. First, RAFT constructs model-compatible supervision through self-conditioned rewriting, semantic filtering, and answer fusion. Second, RAFT performs Answer-Conditioned On-Policy Distillation, where the original instruction-tuned model provides soft targets on student-generated trajectories while being conditioned on the fused answer as helpful context. We further introduce top-K temperature distillation and EMA-based adaptive loss balancing to stabilize the domain-general trade-off. Across three instruction-tuned backbones and five domains, RAFT improves average domain accuracy by 23.2% over standard SFT, while recovering part of the SFT-induced degradation on MS-Bench and IFEval, with relative improvements of 18.2% and 10.2%, respectively. These results show that coupling data refinement with trajectory-level preservation provides an effective recipe for domain fine-tuning with alleviated forgetting.

模型微调知识保持数据重构

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。