模型越大越会记忆事实，而非真正懂语言，该研究提出更小的专用语言模型方案。
Towards Fundamental Language Models: Does Linguistic Competence Scale with Model Size?
- 用小型模型专注语言能力，事实查询交由外部工具处理
- 320亿参数模型内知识增长速度远超语言能力提升
- 适合追求高效、可解释性的实际应用开发者
大语言模型虽具强大语言能力，但存在幻觉、偏见、隐私及高计算成本等局限，主要源于语言能力与事实记忆在单一模型中混合。本文提出并实证支持基础语言模型（FLM）范式：采用小规模、语言能力强的模型，将事实检索交由外部工具完成。我们评估了135M至32B参数模型在语言能力、外部事实知识和内部事实知识三个维度的表现。结果表明，尽管语言能力和事实知识随模型规模提升，但内部事实知识增长显著更快，说明模型大小更关联记忆而非核心语言能力。该发现支持模块化语言建模路径，即以紧凑且语言熟练的模型为基础，结合工具构建系统。FLM范式为更高效、可解释、可持续的自然语言处理提供新方向。
原文摘要 · Abstract (English)
Large Language Models offer impressive language capabilities but suffer from well-known limitations, including hallucinations, biases, privacy concerns, and high computational costs. These issues are largely driven by the combination of linguistic competence and factual memorization within a single monolithic model. This paper introduces and empirically supports the Fundamental Language Model (FLM) paradigm, which advocates for smaller, linguistically competent models that offload factual retrieval to external tools. We evaluate models ranging from 135M to 32B parameters across three dimensions: linguistic competence, external factual knowledge, and internal factual knowledge. Our findings reveal that while both linguistic competence and factual knowledge improve with scale, internal factual knowledge grows significantly faster, suggesting that model size is more closely tied to memorization than to core language ability. These results support a modular approach to language modeling, where compact, linguistically proficient models serve as the foundation for tool-augmented systems. The FLM paradigm offers a path toward more efficient, interpretable, and sustainable NLP solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。