通过分析模型生成文本的预测能力,推断其参数规模下限。
Inferring the Size of Large Language Models From Popular Text Memorization

- 利用经典文本片段的预测准确率,构建模型记忆强度信号
- 在多个开源模型上验证,推断结果与实际参数量高度一致
- 可探测闭源模型的隐藏设计策略,适合关注模型透明度的研究者
主流大语言模型的参数量常被开发者保密,导致模型能力与成本评估缺乏基准。本文提出一种黑箱方法,仅通过提交文本片段并观察下一个词的预测结果,即可推断模型参数量的保守下界。核心思路是:经典文献、宗教文本等广泛存在于预训练数据中,模型对不同长度文本片段的预测准确率反映其记忆程度,而记忆容量受参数总量限制。我们将多种文本和片段长度的预测表现整合为单一精度特征向量,并基于此构建两种互补方法:一对统计检验用于判断两个模型谁更大,以及基于主成分分析(PCA)的缩放定律估计器,从特征向量中提取一维隐变量映射至参数量。在多个开源模型上验证,该方法获得准确可靠的下界。应用于主流闭源模型时,成功揭示内部产品层级,并暴露行业缩放策略差异:部分厂商呈现显著参数增长迹象,另一些则严格控制参数上限,证明即使在接口受限下,也能系统性探测隐藏设计决策。
原文摘要 · Abstract (English)
The parameter counts of the most widely used large language models (LLMs) are often withheld by their developers, leaving model size -- a primary reference point for interpreting capabilities and costs -- largely undisclosed. We propose a black-box method to infer conservative lower bounds on LLM size from generated text outputs alone, requiring nothing beyond the ability to submit text fragments and observe next-token predictions. Our approach is grounded in a key observation: popular, widely-circulated texts -- such as classical literature, religious texts, and foundational documents -- are present in virtually every large-scale pretraining corpus, and how accurately a model predicts the next word across text fragments of varying length is a reliable signal of how much it has memorized them, which in turn is fundamentally limited by its total parameter count. We aggregate this memorization signal across a diverse corpus of texts and fragment lengths into a single accuracy profile vector per model, and build two complementary inference methods on top of it: a pairwise statistical test that determines which of two models is larger, and a scaling-law estimator that extracts a one-dimensional latent index from these vectors via Principal Component Analysis (PCA) to map the aggregated signal to a parameter count. Validated on a broad set of open-weight models, both methods produce accurate and reliable lower bounds. When applied to popular closed-weight models, our framework recovers internal product hierarchies and reveals a clear divergence in industry scaling strategies: while some developers yield significantly higher bounds indicative of large generational parameter growth, others operate under strict parameter ceilings, demonstrating that hidden design choices can be systematically probed even under strict API limitations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。