找出语言模型隐式推理所需的最小参数量，揭示模型与数据复杂度的匹配规律。
Finding the Minimal Parameter Budget for Implicit Reasoning: A Data Complexity Driven Scaling Law for Language Models
- 在模拟知识图谱环境中从零训练模型，通过多跳推理任务测试隐式推理能力。
- 发现最优模型每参数最多可处理0.008比特信息，且该能力与图搜索熵相关。
- 为模型规模选择提供依据，适合关注大模型推理机制的研究者参考。
推理是语言模型的核心能力，但预训练阶段支持推理所需的最小模型容量仍不明确。本文研究隐式推理所需的最小参数预算，即无需显式思维链监督即可从已学知识中推断新事实的能力。我们在受控的合成环境中从头预训练语言模型，该环境模拟真实世界知识图谱的结构与分布，并评估模型通过多跳推理补全缺失边的能力。从理论和实证两方面，我们发现最优参数预算与图搜索熵之间存在缩放规律。在多种模型规模、训练步数和图复杂度下，我们表明最优语言模型每参数最多可可靠处理0.008比特信息。结果刻画了预训练中隐式推理的最小必要容量，为模型规模与数据复杂度的匹配提供了原则性指导，并深化了对大语言模型推理缩放行为的理解。
原文摘要 · Abstract (English)
Reasoning is a core capability of language models (LMs), yet it remains unclear how much model capacity is necessary to support reasoning during pretraining. In this work, we study the minimal parameter budget required for implicit reasoning, defined as the ability to infer new facts from learned knowledge without explicit chain-of-thought supervision. To isolate this phenomenon, we pretrain LMs from scratch in a controlled synthetic environment that mimics the structure and distribution of real-world knowledge graphs, and evaluate their ability to complete missing edges via multi-hop inference. From both a theoretical and an empirical perspective, we identify a scaling law linking this optimal parameter budget to a graph search entropy measure. Across a wide range of model sizes, training steps, and graph complexities, we show that an optimally sized language model can reliably reason over approximately 0.008 bits of information per parameter at most. Our results characterize the minimal sufficient capacity for implicit reasoning during pretraining. Our findings provide principled guidance for matching model size to data complexity and offer new insights into the scaling behavior of reasoning in large language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。