arXiv:2605.18607cs.CLcs.LG2026-05

用专家解题轨迹统计量预测大模型下游性能,更准更快。

Forecasting Downstream Performance of LLMs With Proxy Metrics

论文配图:Forecasting Downstream Performance of LLMs With Proxy Metrics
图 1 · 摘自论文原文
  • 从专家解题路径中提取熵、top-k准确率等统计量作为代理指标。
  • 在模型选型、预训练数据筛选和训练过程预测中,性能超越传统方法。
  • 适合需要快速评估模型能力的研究者和工程团队使用。

语言模型发展常依赖于比较决策:选择何种架构、预训练语料或训练方案。这些决策需可靠性能预测,但现有信号存在根本局限:交叉熵损失与下游能力关联弱,直接下游评估成本高、稀疏且早期阶段信息不足。本文提出通过聚合候选模型在专家撰写解题路径上的下一个词分布的分项统计量(如熵、top-k准确率、专家词排名)构建代理指标。在三个场景中,代理指标均显著优于基于损失和计算量的基线:1)跨家族模型选型中,平均斯皮尔曼等级相关系数达0.81(交叉熵仅0.36);2)预训练数据筛选中,以约1/10,000的计算量可靠排序25个语料库,突破现有方法极限;3)训练期预测中,可对18倍计算量跨度的下游准确率进行外推,误差仅为现有方法的一半。结果表明,专家轨迹是评估模型能力的普适信号源,可贯穿整个模型开发周期实现可靠性能预测。

原文摘要 · Abstract (English)

Progress in language model development is often driven by comparative decisions: which architecture to adopt, which pretraining corpus to use, or which training recipe to apply. Making these decisions well requires reliable performance forecasts, yet the two commonly used signals are fundamentally limited. Cross-entropy loss is poorly aligned with downstream capabilities, and direct downstream evaluation is expensive, sparse, and often uninformative at early training stages. Instead, we propose to construct proxy metrics by aggregating token-level statistics, such as entropy, top-k accuracy, and expert token rank, from a candidate model's next token distribution over expert-written solutions. Across three settings, our proxies consistently outperform loss- and compute-based baselines: 1) For cross-family model selection, they rank a heterogeneous population of reasoning models with mean Spearman Rho = 0.81 (vs. Rho = 0.36 for cross-entropy loss); 2) For pretraining data selection, they reliably rank 25 candidate corpora for a target model at roughly $10{,}000\times$ less compute than direct evaluation, pushing the Pareto frontier beyond existing methods; and 3) for training-time forecasting, they extrapolate downstream accuracy across an $18\times$ compute horizon with roughly half the error of existing alternatives. Together, these results suggest that expert trajectories are a broadly useful source of signal for assessing model capabilities, enabling reliable performance forecasting throughout the model development life cycle.

大模型评估性能预测代理指标

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。