提出新指标揭示大模型预测排序规律,补足传统方法的不足。
Relative-Based Scaling Law for Neural Language Models
- 用相对概率指标衡量正确词在预测排名中的位置
- 实验验证该指标在四数据集四模型族中稳定有效
- 适用于理解模型涌现现象与理论建模
缩放定律旨在准确预测模型在不同规模下的性能。现有研究几乎全部依赖交叉熵作为评估指标,但交叉熵仅反映正确词的绝对概率,忽略了正确与错误词之间的相对排序。而相对排序对语言模型至关重要,尤其在贪心采样场景中。为解决这一局限,本文从相对排序视角研究缩放规律。首先提出相对基础概率(RBP)指标,量化正确词排在前k个预测中的概率。基于此构建了相对基础缩放定律,描述RBP随模型规模增长的变化规律。在四个数据集和四个模型家族上,跨越五个数量级的广泛实验验证了该定律的稳健性与准确性。最后通过两个实例展示其应用价值:深入解释模型涌现现象,并助力发现缩放定律的底层理论。总体而言,相对基础缩放定律补充了交叉熵视角,推动对大语言模型缩放更完整的理解,为实际开发与理论探索提供重要启示。
原文摘要 · Abstract (English)
Scaling laws aim to accurately predict model performance across different scales. Existing scaling-law studies almost exclusively rely on cross-entropy as the evaluation metric. However, cross-entropy provides only a partial view of performance: it measures the absolute probability assigned to the correct token, but ignores the relative ordering between correct and incorrect tokens. Yet, relative ordering is crucial for language models, such as in greedy-sampling scenario. To address this limitation, we investigate scaling from the perspective of relative ordering. We first propose the Relative-Based Probability (RBP) metric, which quantifies the probability that the correct token is ranked among the top predictions. Building on this metric, we establish the Relative-Based Scaling Law, which characterizes how RBP improves with increasing model size. Through extensive experiments on four datasets and four model families spanning five orders of magnitude, we demonstrate the robustness and accuracy of this law. Finally, we illustrate the broad application of this law with two examples, namely providing a deeper explanation of emergence phenomena and facilitating finding fundamental theories of scaling laws. In summary, the Relative-Based Scaling Law complements the cross-entropy perspective and contributes to a more complete understanding of scaling large language models. Thus, it offers valuable insights for both practical development and theoretical exploration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。