arXiv:2608.03709cs.CL2026-08

用训练初期数据预测模型最终表现,提前淘汰无效配置。

Predicting Deep Neural Network Training Outcomes from Early Training Telemetry

论文配图:Predicting Deep Neural Network Training Outcomes from Early Training Telemetry
图 1 · 摘自论文原文
  • 仅用前五轮的损失、梯度等指标预测结果。
  • 准确率预测R²达0.92–0.99,性能排序正确率超98%。
  • 单轮后即可预警失败,适合大规模调参加速。

深度神经网络的大规模超参数搜索耗费大量算力在注定失败的配置上。我们研究单次训练运行自身的早期遥测数据——每轮损失、训练准确率、梯度信噪比、权重范数增长及激活饱和快照——结合采样的超参数,能否在不依赖其他运行的情况下预测其最终结果。评估三个任务:最终测试准确率、域内相对性能、训练动态失败(包括数值发散)。在六种架构/数据集组合的23,788次训练中,仅使用前五个训练周期的遥测数据,基于梯度提升树的回归模型对最终准确率的R²达到0.92–0.99,相对分类的ROC-AUC为0.983–0.998。单轮后即具备有效预测能力。成对消融显示,梯度与权重层面的遥测相比仅靠损失和准确率曲线有统计显著提升,但实际收益因领域而异。相似架构间迁移能力强,跨数据集迁移受限于准确率量级差异,而非关系本质丧失。结果表明,早期遥测可为算力分配提供实用决策支持,并提醒人工监督自动化干预。

原文摘要 · Abstract (English)

Large hyperparameter sweeps for deep neural networks spend substantial compute on configurations that are effectively doomed from the first few epochs. We study whether a single training run's own early telemetry - per-epoch loss, training accuracy, gradient signal-to-noise ratio, weight-norm growth, and an activation-saturation snapshot - together with its sampled hyperparameters, can predict that run's eventual outcome without reference to other runs. We evaluate three prediction tasks: final test accuracy, relative performance within a domain, and training-dynamics failure, including numerical divergence. Across 23,788 training runs spanning six architecture/dataset combinations, gradient-boosted trees using only the first five epochs of telemetry achieve R^2 = 0.92-0.99 for final-accuracy regression and ROC-AUC = 0.983-0.998 for relative classification on a permanently held-out set of hyperparameter configurations. Useful prediction is already available after a single epoch. A paired ablation shows that gradient- and weight-level telemetry provides a statistically consistent improvement over loss and accuracy curves alone, although the practical gain varies by domain. Transfer is strong between similar architectures, while cross-dataset transfer is limited mainly by differences in accuracy scale rather than loss of the underlying relationship. These results suggest that early-training telemetry can provide a practical decision-support signal for compute allocation while motivating human oversight for any automated intervention.

超参优化早期预测训练监控高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。