用关键标记路由解决大模型领域适配中的遗忘问题
CPR for LLMs: Critical-Point Routing against Catastrophic Forgetting in Domain Adaptation
- 按关键词粒度切换基础模型与专家模型，避免能力冲突
- 领域性能提升1.4-5.5%，通用能力损失压至0.5%以内
- 适合需兼顾通用性与领域专精的场景，如医疗、法律问答
监督微调（SFT）是将大语言模型适配到目标领域的主流方法，但常导致模型通用能力下降，即灾难性遗忘。现有方法多通过修改SFT损失缓解遗忘，却难以摆脱领域性能与通用性的权衡。本文提出CPR（关键点路由），在模型层面解耦两种能力：保留原始基础模型以维持通用性，仅在需要领域知识时调用微调专家模型。CPR基于关键标记（即基础模型失败但专家成功的位置）进行逐标记路由，训练轻量级分层路由器估算每标记调用专家的概率，并结合动量平滑与阈值门控的推理机制。在多种模型-领域配置下，CPR均达当前最优表现：领域性能优于纯SFT专家模型1.4-5.5%，通用能力损失由3.4-14.5%降至最多0.5%，且仅约1/3标记需调用专家。
原文摘要 · Abstract (English)
Supervised fine-tuning (SFT) is the de facto standard for adapting large language models (LLMs) to target domains, but it often degrades the model's general capabilities, a phenomenon known as catastrophic forgetting. Existing approaches typically modify the SFT loss to mitigate forgetting, but they inevitably operate along a domain-generality trade-off. In this work, we step outside this trade-off by decoupling the two capabilities at the model level: we keep the original base model for general capability, and selectively invoke the SFT expert only when domain-specific knowledge is required. Specifically, we propose CPR (Critical-Point Routing), a token-level routing framework between a base model and its expert derivative, based on critical tokens where the base model fails but the expert succeeds. We train a lightweight hierarchical router that estimates the expert-call probability per token, and pair it with a tailored inference procedure that combines momentum smoothing and threshold gating. Across diverse model-domain configurations, CPR achieves state-of-the-art across all settings, surpassing SFT expert by 1.4-5.5% in domain performance while recovering its general-capability drop from 3.4-14.5% to at most 0.5%, with minimal overhead from invoking the expert on only one-third of tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。