arXiv:2608.20794cs.AI2026-08

让大模型在微调后仍能准确说出事实,避免‘知道却说不出来’的问题。

Knowing but Not Saying: Preventing Factual Access Failures in LLM SFT via Recall-Anchored Distillation

论文配图:Knowing but Not Saying: Preventing Factual Access Failures in LLM SFT via Recall-Anchored Distillation
图 1 · 摘自论文原文
  • 用原始模型的生成分布做自蒸馏,保持跨领域事实表达能力。
  • 在医学问答任务上,恢复了约30%丢失的跨域召回率。
  • 无需标注数据或外部评分,适合医疗、法律等高可靠性场景。

监督微调(SFT)可能导致模型在目标领域外的事实表现下降。这种下降常被归因于灾难性遗忘,但开放式事实错误未必意味着知识被抹除。本文识别出一种更具体的现象——事实访问失败:微调后模型虽能在受限评估中识别正确答案,却无法在闭卷生成中输出。通过基准对比、同事实多选与生成探测,发现退化既包括错误生成,也包含冗余、格式不匹配等表达问题。为此提出回忆锚定蒸馏(RAD),通过对未标记的域外文本,将适配模型与原始基模型的软延续分布对齐,保留域外生成能力。RAD无需黄金答案、外部评价或标注数据。在三个骨干模型上微调医学问答数据集MedMCQA,RAD一致恢复了部分丢失的域外召回率,同时保持目标领域适应性。相比仅重复暴露域外文本,证明关键信号是基模型的软分布而非额外文本本身。

原文摘要 · Abstract (English)

Supervised fine-tuning (SFT) can degrade factual behavior outside the target domain. This degradation is often described as catastrophic forgetting, yet open-ended factual failures do not necessarily imply that the underlying facts have been erased. In this work, we identify a more specific phenomenon, factual access failure: after domain SFT, models can still recognize or rank the correct answer under constrained evaluation, while failing to produce it in closed-book generation. Through benchmark-level comparisons, same-fact multiple-choice and generation probes, and failure-mode analysis, we show that SFT-induced factual degradation reflects both genuine wrong-answer generations and expression-level failures such as verbosity, formatting mismatch, and exact-match artifacts. To address this problem, we introduce Recall-Anchored Distillation (RAD), a base-anchored self-distillation objective that preserves out-of-distribution generation behavior by aligning the adapted model with the original base model's soft continuation distribution on unlabeled OOD text. RAD requires no gold OOD answers, external judges, or labeled factual data. Across three backbones fine-tuned on MedMCQA, RAD recovers a consistent portion of the lost OOD recall while preserving target-domain adaptation. Compared with replay on the same OOD text, RAD shows that the key preservation signal is the base model's soft distribution rather than additional text exposure alone.

大模型微调事实生成自蒸馏

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。