arXiv:2412.15321cs.CV2024-12AAAI被引 30

用块预测替代逐像素生成,显著降低训练成本。

Next Patch Prediction for Autoregressive Visual Generation

  • 将图像分块后预测下一区块,提升信息密度与效率
  • 训练成本降至0.6倍,图像质量提升1.0 FID得分
  • 无需修改模型结构,可直接接入现有生成系统

基于自回归范式的视觉生成模型在融合语言与视觉任务方面展现出巨大潜力。本文重新思考自回归图像生成中的下一个标记预测(NTP)机制,提出新颖的下一个块预测(NPP)范式。核心思想是将图像标记分组聚合为信息密度更高的块标记。通过以更紧凑的块序列作为输入,模型学习预测下一个块,显著降低计算开销。为进一步利用图像数据的天然层次结构,提出多尺度从粗到细的块分组策略:训练从大块开始,逐步细化至1×1块,保持原始推理过程不变。在多种模型规模下的实验表明,该方法可将训练成本降至约0.6倍,同时在ImageNet 256x256生成基准上,图像质量最高提升1.0 FID分数。方法无需修改原有自回归模型架构,不引入额外可训练参数,也不需设计专用图像分词器,是一种灵活、即插即用的视觉生成增强方案。

原文摘要 · Abstract (English)

Autoregressive models, built based on the Next Token Prediction (NTP) paradigm, show great potential in developing a unified framework that integrates both language and vision tasks. Pioneering works introduce NTP to autoregressive visual generation tasks. In this work, we rethink the NTP for autoregressive image generation and extend it to a novel Next Patch Prediction (NPP) paradigm. Our key idea is to group and aggregate image tokens into patch tokens with higher information density. By using patch tokens as a more compact input sequence, the autoregressive model is trained to predict the next patch, significantly reducing computational costs. To further exploit the natural hierarchical structure of image data, we propose a multi-scale coarse-to-fine patch grouping strategy. With this strategy, the training process begins with a large patch size and ends with vanilla NTP where the patch size is 1$\times$1, thus maintaining the original inference process without modifications. Extensive experiments across a diverse range of model sizes demonstrate that NPP could reduce the training cost to around 0.6 times while improving image generation quality by up to 1.0 FID score on the ImageNet 256x256 generation benchmark. Notably, our method retains the original autoregressive model architecture without introducing additional trainable parameters or specifically designing a custom image tokenizer, offering a flexible and plug-and-play solution for enhancing autoregressive visual generation.

自回归生成图像生成高效训练块预测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。