通过微调预测大模型新能力何时出现，提前掌握模型进化节点。
Predicting Emergent Capabilities by Finetuning
- 用不同数据量微调小模型，找到能力涌现的临界点。
- 在四个NLP基准上验证，能准确预测4倍计算量下是否出现新能力。
- 适合想预判未来模型性能的研究者和工程团队使用。
现代大语言模型扩展面临的核心挑战之一是难以理解新兴能力。尽管预训练损失随算力变化可预测，但下游任务表现却常呈现突现式跃升，难以预判未来模型的能力。本文提出‘涌现预测’任务：基于当前模型在某任务上的随机少量样本准确率，能否预测未来模型（如GPT-N+1）是否具备非平凡准确率？研究发现，对模型进行特定任务微调，可使涌现发生所需的算力阈值向更小模型偏移。通过在不同数据量下微调小模型并拟合参数化函数，可建立‘涌现规律’。在MMLU、GSM8K、CommonsenseQA和CoLA四个标准NLP基准上验证，仅使用小规模模型即可准确预测出计算量增加4倍后是否出现能力跃升。最后展示了两个实际应用场景，证明该方法对模型规划具有实用价值。
原文摘要 · Abstract (English)
A fundamental open challenge in modern LLM scaling is the lack of understanding around emergent capabilities. In particular, language model pretraining loss is known to be highly predictable as a function of compute. However, downstream capabilities are far less predictable -- sometimes even exhibiting emergent jumps -- which makes it challenging to anticipate the capabilities of future models. In this work, we first pose the task of emergence prediction: given access to current LLMs that have random few-shot accuracy on a task, can we predict whether future models (GPT-N+1) will have non-trivial accuracy on that task? We then discover a simple insight for this problem: finetuning LLMs on a given task can shift the point in scaling at which emergence occurs towards less capable models. To operationalize this insight, we can finetune LLMs with varying amounts of data and fit a parametric function that predicts when emergence will occur (i.e., "emergence laws"). We validate this approach using four standard NLP benchmarks where large-scale open-source LLMs already demonstrate emergence (MMLU, GSM8K, CommonsenseQA, and CoLA). Using only small-scale LLMs, we find that, in some cases, we can accurately predict whether models trained with up to 4x more compute have emerged. Finally, we present a case study of two realistic uses for emergence prediction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。