发现扩散模型训练的最优算力规律，建议多用数据而非大模型。
Abra: Scaling Diffusion Image Training
- 在10^19到10^22 FLOPs间系统测试扩散模型，构建算力缩放规律。
- 最优训练需约200张图像令牌/参数，是语言模型的10倍。
- 模型对过拟合不敏感，应优先增加数据量而非扩大模型规模。
计算最优缩放定律指导前沿语言模型训练，但在视觉生成领域仍基本未被探索。本文通过Abra——一个在三个数量级算力范围（10^19至10^22 FLOPs）上训练的流匹配变压器系列——对文本到图像扩散模型进行了系统的缩放定律研究，达到远超以往工作的计算预算。我们证明，扩散模型的缩放规律与语言模型一样可预测，但需更多数据才能实现最优训练：计算最优发生在每参数约200个图像令牌，是语言模型Chinchilla最优方案的十倍。我们还发现，与语言模型不同，扩散模型对过训练具有鲁棒性，从业者应倾向使用更多数据而非更大模型。最后，这种可预测性不仅体现在训练损失上，还扩展至生成质量指标、最优引导尺度（CFG）、表征质量乃至训练曲线形态，所有曲线均可归一化为统一形式。
原文摘要 · Abstract (English)
Compute-optimal scaling laws guide the training of frontier language models yet remain largely unexplored for visual generation. We present a systematic scaling law study for text-to-image diffusion models using Abra, a controlled family of flow-matching transformers trained across three orders of magnitude worth of compute ($10^{19}$ to $10^{22}$ FLOPs), reaching significantly larger compute budgets than previous works. We demonstrate that diffusion models scale just as predictably as language models but require far more data to train optimally: compute optimality occurs at approximately $200$ image tokens per parameter, ten times the Chinchilla compute-optimal prescription for LLMs. We show that unlike language models, diffusion models are robust to overtraining and that practitioners should err on the side of more data rather than a larger model. Finally, we show that this predictability extends beyond training loss to generative quality metrics, optimal CFG settings, representation quality, and even the shape of the training curves, which collapse onto a universal form.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。