用轻量级测试区分大模型真学知识还是只模仿语言
Beyond Perplexity: A Lightweight Benchmark for Knowledge Retention in Supervised Fine-Tuning
- 设计对比样例测试模型对事实的掌握程度
- 发现语言风格收敛快但知识留存慢
- 适合调试微调过程、评估模型真实能力
监督微调（SFT）是向大语言模型注入领域知识的标准方法。然而，仅依赖验证困惑度监控训练往往不足，因为它混淆了风格模仿与真实事实内化。为此，我们提出知识保留（KR-Test）测试框架，一种轻量、基于语料库的评估方法，可区分事实学习与语言特征。KR-Test通过自动生成对比样本，测量模型对正确与错误延续的似然偏好，无需指令微调或生成解码。我们通过“盲测与专家”基线分析验证了框架的可靠性。此外，利用该框架分析低秩适配（LoRA）的训练动态，揭示了语言收敛与知识留存之间的细粒度脱节，提升了微调过程的可解释性。
原文摘要 · Abstract (English)
Supervised Fine-Tuning (SFT) is a standard approach for injecting domain knowledge into Large Language Models (LLMs). However, relying on validation perplexity to monitor training is often insufficient, as it confounds stylistic mimicry with genuine factual internalization. To address this, we introduce the Knowledge Retention (KR) Test , a lightweight, corpus-grounded evaluation framework designed to distinguish factual learning from linguistics. KR-Test utilizes automatically generated contrastive examples to measure likelihood preferences for correct versus incorrect continuations, requiring no instruction tuning or generative decoding. We validate the framework's integrity through a "blind vs. oracle" baseline analysis. Furthermore, we demonstrate the diagnostic capabilities of KR-Test by analyzing the training dynamics of Low-Rank Adaptation (LoRA). By exposing the fine-grained dissociation between linguistic convergence and knowledge retention, KR-Test enhances the interpretability of fine-tuning dynamics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。