探索像素级生成预训练的扩展规律，发现算力是主要瓶颈。
Rethinking Generative Image Pretraining: How Far Are We From Scaling Up Next-Pixel Prediction?
- 用Transformer在32x32图像上做逐像素预测，研究不同任务下的最优扩展策略。
- 生成任务需数据量比分类任务快3至5倍增长，且模型规模随分辨率提升需更快增长。
- 算力每年增4-5倍，未来五年有望实现完整像素级图像建模。
本文研究自回归逐像素预测的扩展特性，这是一种简单、端到端但未被充分探索的统一视觉模型框架。从32x32分辨率图像开始，我们在计算预算达7×10¹⁹ FLOPs的条件下，使用等浮点运算量（IsoFlops）配置训练一系列Transformer，并评估三项指标：逐像素预测目标、ImageNet分类准确率以及基于Fréchet距离的生成补全效果。首先，最优扩展策略高度依赖任务：在固定32x32分辨率下，生成任务的最优设置要求数据量比分类任务快3至5倍增长。其次，随着图像分辨率提高，模型规模必须远快于数据量增长。出人意料的是，通过外推发现，主要瓶颈是算力而非训练数据量。鉴于算力每年增长4至5倍，我们预测未来五年内实现像素级图像建模将成为可能。
原文摘要 · Abstract (English)
This paper investigates the scaling properties of autoregressive next-pixel prediction, a simple, end-to-end yet under-explored framework for unified vision models. Starting with images at resolutions of 32x32, we train a family of Transformers using IsoFlops profiles across compute budgets up to 7e19 FLOPs and evaluate three distinct target metrics: next-pixel prediction objective, ImageNet classification accuracy, and generation-based completion measured by Fr'echet Distance. First, optimal scaling strategy is critically task-dependent. At a fixed resolution of 32x32 alone, the optimal scaling properties for image classification and image generation diverge, where generation optimal setup requires the data size grow three to five times faster than for the classification optimal setup. Second, as image resolution increases, the optimal scaling strategy indicates that the model size must grow much faster than data size. Surprisingly, by projecting our findings, we discover that the primary bottleneck is compute rather than the amount of training data. As compute continues to grow four to five times annually, we forecast the feasibility of pixel-by-pixel modeling of images within the next five years.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。