在消费级显卡上，训练时间越长，该用多大的模型才最好？
Time is Not Compute: Scaling Laws for Wall-Clock Constrained Training on Consumer GPUs
- 在固定时间内，模型太小会过拟合，太大则欠训练，存在最优尺寸。
- 模型最佳大小随时间增长呈幂律关系，指数为0.60，比计算最优更快增长。
- 短时间受算力限制，长时间受数据限制，中间阶段无明显最优点。
缩放定律将模型质量与计算预算（FLOPs）关联，但实践者面临的是墙钟时间约束，而非计算预算。我们在消费级显卡（RTX 4090）上，针对5分钟至24小时的固定时间预算，研究了模型尺寸的最优选择。涵盖5000万至10.31亿参数的70多次实验表明：（1）每个时间预算下均出现倒U型曲线，过小模型过拟合，过大模型欠训练；（2）最优模型规模满足 $N^* /propto t^{0.60}$，增长速度超过Chinchilla的 $C^{0.50}$，$α=0.60 \± 0.07$ 在所有敏感性分析中均显著优于计算最优；（3）双倒U型机制：短时间下由算力瓶颈导致倒U形，长时间下由数据瓶颈（过拟合）引发倒U形，中间存在短暂无倒U形的过渡区。这些发现对在消费硬件上训练的研究者具有直接指导意义，因墙钟时间才是关键约束。我们开源全部代码、日志及70+实验配置。
原文摘要 · Abstract (English)
Scaling laws relate model quality to compute budget (FLOPs), but practitioners face wall-clock time constraints, not compute budgets. We study optimal model sizing under fixed time budgets from 5 minutes to 24 hours on consumer GPUs (RTX 4090). Across 70+ runs spanning 50M--1031M parameters, we find: (1)~at each time budget a U-shaped curve emerges where too-small models overfit and too-large models undertrain; (2)~optimal model size follows $N^* \propto t^{0.60}$, growing \emph{faster} than Chinchilla's $N^* \propto C^{0.50}$, with $α= 0.60 \pm 0.07$ robustly exceeding compute-optimal across all sensitivity analyses; (3)~a \emph{dual U-shape mechanism}: short-budget U-curves arise from compute bottlenecks, while long-budget U-curves emerge from data bottlenecks (overfitting), with an intermediate regime where the U-curve temporarily disappears. These findings have immediate implications for researchers training on consumer hardware, where wall-clock time -- not FLOPs -- is the binding constraint. We release all code, logs, and 70+ experimental configurations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。