提出新方法预测大模型知识保留上限，无需训练即可评估。
Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Training
- 用信息论构建SMI模型，融合知识频率、特异性与模型规模。
- 在24个模型上验证，10亿参数以上模型预测准确率$R^2>0.7$。
- 发现训练数据和模型规模的增益有上限，适合想优化预训练的团队。
本文针对大模型预训练中知识保留能力的量化难题，提出一种基于信息论的预测方法——规模依赖互信息（SMI）。SMI综合考虑知识频率、知识特异性和模型规模，可预先估算闭卷问答准确率。通过在21个公开及3个定制模型的预训练语料上进行大规模文档检索，并结合多模板问答评估，实验表明：当模型参数量超过10亿时，SMI预测准确率$R^2 > 0.7$，显著优于基于重复性的基线方法。分析揭示模型与数据规模扩展存在边际递减效应，证明仅靠预训练存在内在知识保留上限，提示需引入检索等增强策略。代码与数据已开源。
原文摘要 · Abstract (English)
The GPT-4 technical report suggests that downstream performance can be predicted from pre-training signals, but offers little methodological detail on how to quantify this. This work address this gap by modeling knowledge retention, the capacity of a pre-trained language model to memorize factual information from its corpus, and introduce a principled method to estimate it prior to training. We propose Size-dependent Mutual Information (SMI), an information-theoretic predictor that integrates knowledge frequency, knowledge specificity, and model size to forecast closed-book question answering (QA) accuracy. SMI is validated through large-scale document retrieval over the disclosed pre-training corpora of 21 public and 3 custom models, combined with a robust multi-template QA evaluation. Experiments show that SMI significantly outperforms repetition-based baselines and achieves $R^2$ > 0.7 in predicting QA accuracy for models above 1B parameters, without additional training. The analysis further reveals diminishing returns from scaling data and model size and provides evidence for an intrinsic upper bound on knowledge retention achievable by pre-training alone, motivating retrieval and other augmentation strategies. The dataset and code are available at https://github.com/yuhui1038/SMI.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。