让模型在专精领域提升性能的同时，不丢失通用能力。
Self-Specialized Teachers for Domain Post-Training
- 用自生成前缀蒸馏，让模型在专精任务中学习而不遗忘通用知识。
- 在金融、医疗、法律任务上，通用评估得分提升4.8~5.0分。
- 无需外部教师或通用数据，适合资源有限的领域适配场景。
仅针对目标域进行后训练可提升特定领域表现，但会损害通用基础模型原有的能力。本文研究在仅有目标域数据而无代表性回放语料时的这一问题。提出自专精教师蒸馏（SSTD），一种两阶段方法：首先将基础模型复制并训练为领域教师，再利用学生自身采样的前缀对齐其输出分布。教师训练结合目标监督、基础模型感知的关键词加权及与冻结基础模型的分布对齐；在线蒸馏则将领域反馈作用于推理时可能遇到的状态。在金融数值推理、医疗问答和法律裁决识别任务上，SSTD在保持直接微调的领域提升的同时，通用评测集平均得分提高4.8–5.0分（报告运行点）。该效果在Qwen3不同规模及Gemma骨干网络上均成立。SSTD无需外部教师或通用回放数据。
原文摘要 · Abstract (English)
Target-only post-training can improve performance in a specialized domain while degrading behaviors that a general-purpose base model acquired before adaptation. We study this problem when target-domain data are available but a representative replay corpus is not. We propose self-specialized teacher distillation (SSTD), a two-stage procedure that first trains a copy of the base model into a domain teacher, then distills its token distribution to a student on prefixes sampled from the student itself. Teacher training combines standard target supervision with base-aware key-token weighting and distribution alignment to the frozen base model; on-policy distillation then places domain feedback on states the student can encounter at inference time. On financial numerical reasoning, medical question answering, and legal holding identification, SSTD retains much of the target improvement of direct fine-tuning while improving the mean score on the evaluated general suite by 4.8--5.0 points at the reported operating point. The pattern persists across Qwen3 sizes and on Gemma backbones. SSTD requires neither an external teacher nor general replay data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。