arXiv:2602.12005cs.CL2026-02

小模型该学什么不该学,不能只看损失值。

LaCy: What Small Language Models Can and Should Learn is Not Just a Question of Loss

  • 用损失值+语法信息判断小模型该自己预测还是调用大模型
  • 在维基类文本上,决策准确率显著提升,生成事实正确性更高
  • 方法简单高效,适合资源有限的场景使用

语言模型持续增大以压缩更多世界知识,但其预训练知识上限受参数量限制。小语言模型(SLMs)能力有限,常生成错误内容。通常通过接入外部资源(如大模型、文档)缓解问题。本文研究:小模型在预训练中应学习哪些词、哪些应通过<CALL>标记委托给外部。发现仅靠损失值无法准确判断:高损失词可能对应合理替代延续,不应触发调用。因此,仅依赖损失不足以判定可学性,需结合句子中词的角色等领域信号。在维基类数据上,引入spaCy轻量语法信息后,显著改善了委托决策。基于此提出LaCy方法,融合损失与事实性信号,指导小模型何时学习、何时求助。实验表明,LaCy模型在与大模型级联生成时,事实得分更高,优于Rho或LLM-judge训练的小模型,且更简单经济。

原文摘要 · Abstract (English)

Language models have consistently grown to compress more world knowledge into their parameters, but the knowledge that can be pretrained into them is upper-bounded by their parameter size. Especially the capacity of Small Language Models (SLMs) is limited, leading to factually incorrect generations. This problem is often mitigated by giving the SLM access to an outside source: the ability to query a larger model, documents, or a database. Under this setting, we study the fundamental question of \emph{which tokens an SLM can and should learn} during pretraining, versus \emph{which ones it should delegate} via a \texttt{<CALL>} token. We find that this is not simply a question of loss: although the loss is predictive of whether a predicted token mismatches the ground-truth, it is insufficient for identifying which predictions would actually lead to factual or semantically invalid continuations. Some high-loss tokens correspond to \emph{acceptable} alternative continuations of a pretraining document and therefore should not trigger a \texttt{<CALL>}. This suggests that learnability cannot be characterized from loss alone, but requires additional domain-specific signals about the role of a token in the sentence. In Wikipedia-like domains, we show that augmenting the loss signal with lightweight grammatical information from a spaCy parser substantially improves delegation decisions. Based on this insight, we propose LaCy, a novel pretraining method that combines loss with factuality signals to decide which tokens an SLM should learn. Our experiments demonstrate that LaCy models successfully learn which tokens to predict and when to call for help. This results in higher FactScores when generating in a cascade with a bigger model and outperforms Rho or LLM-judge trained SLMs, while being simpler and cheaper.

小模型知识蒸馏生成质量轻量化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。