用预训练指标预测大模型微调效果，发现困惑度不可靠，新指标可提升预测准确率超50%。
Can Pre-training Indicators Reliably Predict Fine-tuning Outcomes of LLMs?
- 将模型选择转为成对分类任务，比较不同预训练版本的微调表现
- 在50个10亿参数模型上测试，传统困惑度预测效果差，新指标误差降低超50%
- 适用于优化特定下游任务的预训练方案设计，实用性强
尽管困惑度等预训练阶段指标在规模定律研究中与模型性能高度相关，但在固定模型规模下其预测能力仍不明确，限制了模型选型与开发效率。为此，我们提出将预训练检查点选择问题建模为成对分类任务：预测两个在预训练上不同的大模型，哪个在监督微调（SFT）后表现更好。我们构建了一个数据集，包含50个10亿参数的大模型变体，其预训练配置系统性变化（如目标函数或数据），并在多种下游任务上评估其微调表现。研究发现，传统困惑度是误导性指标。为此，我们引入新的无监督和有监督代理指标，基于预训练阶段信息，成功将相对性能预测误差率降低超过50%。尽管任务本身复杂，但所提代理指标在特定场景下展现出实际应用价值，为面向下游任务优化的预训练设计提供了新路径。
原文摘要 · Abstract (English)
While metrics available during pre-training, such as perplexity, correlate well with model performance at scaling-laws studies, their predictive capacities at a fixed model size remain unclear, hindering effective model selection and development. To address this gap, we formulate the task of selecting pre-training checkpoints to maximize downstream fine-tuning performance as a pairwise classification problem: predicting which of two LLMs, differing in their pre-training, will perform better after supervised fine-tuning (SFT). We construct a dataset using 50 1B parameter LLM variants with systematically varied pre-training configurations, e.g., objectives or data, and evaluate them on diverse downstream tasks after SFT. We first conduct a study and demonstrate that the conventional perplexity is a misleading indicator. As such, we introduce novel unsupervised and supervised proxy metrics derived from pre-training that successfully reduce the relative performance prediction error rate by over 50%. Despite the inherent complexity of this task, we demonstrate the practical utility of our proposed proxies in specific scenarios, paving the way for more efficient design of pre-training schemes optimized for various downstream tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。