让大模型只在相关领域激活微调,保持通用能力不下降。
When to Adapt: Conditional Memory Adapters for Retention-Preserving Domain Specialization

- 用条件记忆机制选择性地注入微调信息,仅对相关输入生效。
- 在医学和新闻任务上提升性能,外推表现保留99.4%~100.1%。
- 适合需要保留通用能力的领域定制场景,如医疗、法律等。
部署于特定领域的大型语言模型需提升领域内性能,同时不损害通用能力。现有参数高效微调方法通常始终启用:其学习到的扰动对所有输入都生效,可能导致域外(OOD)性能下降。我们提出Engram Adapter,将预训练阶段的条件记忆重新用于冻结大模型的后处理适配器。该方法通过局部n-gram模式的多通道匹配与显式占用追踪,构建轻量级选择性先验,使残差注入更可能发生在领域内输入上;同时,一个可学习的标量门控抑制非连贯的域外检索。我们在Qwen3-4B和Qwen3-8B上以AG-News和MedMCQA为适应任务,评估了涵盖推理、翻译、代码生成和法律推理的多个域外基准。Engram Adapter在提升领域内准确率的同时,保持了99.4%–100.1%的平均域外性能;在LegalBench上,其平均表现甚至略优于冻结基线模型,而对比的始终启用基线则显著退化。机制分析表明,尽管域外激活非零,但门控与投影衰减使残差降至隐藏状态范数的约0.08%,导致极小的KL散度与可忽略的准确率变化。结果表明,条件激活是实现模块化、保留能力的领域专化的一种有前景路径。
原文摘要 · Abstract (English)
Large language models deployed in specialized domains must improve in-domain performance without sacrificing general capabilities. Existing parameter-efficient fine-tuning methods are typically always on: their learned perturbations are applied to every input, which can degrade out-of-domain (OOD) performance. We propose Engram Adapter, a framework that repurposes pretraining-time conditional memory as a post-hoc adapter for frozen LLMs. It uses multi-channel matching over local n-gram patterns with explicit occupancy tracking as a lightweight selectivity prior, making residual injection more likely on in-domain inputs while a learned scalar gate suppresses incoherent OOD retrievals. We evaluate on Qwen3-4B and Qwen3-8B with AG-News and MedMCQA as adaptation tasks and OOD benchmarks spanning reasoning, translation, code generation, and legal reasoning. Engram Adapter improves in-domain accuracy while preserving 99.4%--100.1% of average OOD performance; on LegalBench it slightly exceeds the frozen base model on average, whereas comparable always-on baselines degrade sharply. Mechanistic analyses show that although OOD activations are non-zero, gate and projection attenuation reduce residuals to approximately 0.08% of hidden-state norm, yielding small KL drift and negligible accuracy change. These results suggest conditional activation is a promising route toward modular, retention-preserving domain specialization over frozen backbones.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。