用对数似然向量绘制千余模型地图，高效比较语言模型差异。
Mapping 1,000+ Language Models via the Log-Likelihood Vector
- 以文本集上的对数似然向量作模型特征，理论基础扎实。
- 计算成本线性增长，处理超1000个模型仅需数小时。
- 适合大规模模型对比与可视化分析，尤其适用于研究者。
为在大规模下比较自回归语言模型，我们提出使用预定义文本集上计算的对数似然向量作为模型特征。该方法具有坚实的理论基础：当将这些向量视为模型坐标时，其平方欧氏距离可近似文本生成概率的相对熵（Kullback-Leibler divergence）。该方法高度可扩展，计算成本随模型数量和文本样本数呈线性增长，且实现简单，因所需特征可直接从交叉熵损失中获得。我们将该方法应用于超过1,000个语言模型，构建了“模型地图”，为大规模模型分析提供了新视角。
原文摘要 · Abstract (English)
To compare autoregressive language models at scale, we propose using log-likelihood vectors computed on a predefined text set as model features. This approach has a solid theoretical basis: when treated as model coordinates, their squared Euclidean distance approximates the Kullback-Leibler divergence of text-generation probabilities. Our method is highly scalable, with computational cost growing linearly in both the number of models and text samples, and is easy to implement as the required features are derived from cross-entropy loss. Applying this method to over 1,000 language models, we constructed a "model map," providing a new perspective on large-scale model analysis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。