用潜在变量建模大模型性能，揭示不同家族的共性规律。
A Latent Variable Framework for Scaling Laws in Large Language Models
- 引入潜在变量捕捉各模型家族的共同特征
- 在12个基准上验证了性能预测能力
- 适合研究模型家族差异与跨任务泛化
我们提出一种基于潜在变量建模的统计框架，用于分析大语言模型（LLM）的缩放定律。随着大量新架构和训练策略的涌现，以及越来越多基准的出现，不同模型家族间表现出显著异质性，单一全局缩放曲线难以刻画其性能变化。为此，我们构建了一个潜在变量框架，每个模型家族对应一个潜在变量，反映其内在共性特征；模型在不同基准上的表现由该家族的潜在技能决定，而潜在技能又由潜在变量与模型可观测特征共同决定。我们设计了相应的估计方法并证明其统计性质，还开发了高效数值算法以支持估计与下游任务。实证上，我们在 Open LLM Leaderboard（v1/v2）中的12个常用基准上评估了该方法。
原文摘要 · Abstract (English)
We propose a statistical framework built on latent variable modeling for scaling laws of large language models (LLMs). Our work is motivated by the rapid emergence of numerous new LLM families with distinct architectures and training strategies, evaluated on an increasing number of benchmarks. This heterogeneity makes a single global scaling curve inadequate for capturing how performance varies across families and benchmarks. To address this, we propose a latent variable modeling framework in which each LLM family is associated with a latent variable that captures the common underlying features in that family. An LLM's performance on different benchmarks is then driven by its latent skills, which are jointly determined by the latent variable and the model's own observable features. We develop an estimation procedure for this latent variable model and establish its statistical properties. We also design efficient numerical algorithms that support estimation and various downstream tasks. Empirically, we evaluate the approach on 12 widely used benchmarks from the Open LLM Leaderboard (v1/v2).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。