用模型大小、批量和更新次数预测大模型训练损失，可优化资源分配。
Predicting Large Model Test Losses with a Noisy Quadratic System
- 基于模型规模、批量和更新次数构建噪声二次系统预测损失。
- 在计算预算扩展1000倍时仍优于Chinchilla模型的预测效果。
- 适合需要在时间、内存、算力约束下找最优配置的研究者。
我们提出一个预测模型，可根据模型规模（N）、批量大小（B）和权重更新次数（K）估算大模型的预训练损失。这是首个能处理变化批量的损失预测模型。该模型在外推计算预算（最高达1000倍）下的损失预测上优于Chinchilla的损失模型（基于批量和总词元数）。模型可用于在显式或复合资源约束（如时间、内存、算力）下寻找最优的N、B、K配置。实验表明，模型选出的配置接近真实最优解。本工作主张以损失预测替代日益复杂的启发式定律。代码已开源：https://github.com/chuningxdy/Noisy-Quadratic-System。
原文摘要 · Abstract (English)
We introduce a predictive model that estimates the pre-training loss of large models from model size (N), batch size (B) and number of weight updates (K). This is the first loss prediction model that can handle changing batch size. The model outperforms Chinchilla's loss model, a model of the test loss using the batch size and number of tokens, in terms of projecting the loss at extrapolated compute budgets (up to 1000 folds). A natural use of the model is to find optimal N, B, K configurations under explicit and compound resource constraints like time, memory and compute. In our experiments, the model-selected configurations are close to ground-truth optimal. Our work advocates for loss prediction as a better alternative to heuristic-based laws, which are growing in complexity. The implementation is available on https://github.com/chuningxdy/Noisy-Quadratic-System.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。