提出可预测任意视觉单元的自回归框架,提升生成质量与速度。
Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generation
- 用连续回归替代离散分类,支持多种粒度的预测单元
- 在ImageNet-256上以172M模型超越675M模型,推理快20倍
- 无需预训练模块或复杂采样策略,适合高效图像生成场景
自回归建模依赖于逐标记预测范式,广泛应用于先进语言与视觉生成模型。传统上,'标记'被视为最小预测单元,如语言中的离散符号或视觉中的量化图像块。然而,二维图像结构的最佳标记定义仍不明确。此外,自回归模型存在暴露偏差问题,训练时使用教师强制导致推理时错误累积。本文提出xAR,一种广义自回归框架,将标记扩展为实体X,可表示单个图像块、邻域块组($k imes k$)、非局部远距离块组、多尺度(粗到细分辨率)乃至整张图像。同时,将离散标记分类重构为连续实体回归,每步采用流匹配方法。训练时基于噪声实体而非真实标记,实现噪声上下文学习,有效缓解暴露偏差。结果表明,xAR具备两大优势:(1) 支持灵活的预测粒度与空间结构捕捉;(2) 通过避免教师强制显著减轻暴露偏差。在ImageNet-256生成基准上,基础模型xAR-B(172M)性能优于DiT-XL/SiT-XL(675M),推理速度提升20倍;xAR-H达到新最优FID 1.24,运行速度比前驱快2.2倍,且不依赖视觉基础模型(如DINOv2)或高级引导间隔采样。
原文摘要 · Abstract (English)
Autoregressive (AR) modeling, known for its next-token prediction paradigm, underpins state-of-the-art language and visual generative models. Traditionally, a ``token'' is treated as the smallest prediction unit, often a discrete symbol in language or a quantized patch in vision. However, the optimal token definition for 2D image structures remains an open question. Moreover, AR models suffer from exposure bias, where teacher forcing during training leads to error accumulation at inference. In this paper, we propose xAR, a generalized AR framework that extends the notion of a token to an entity X, which can represent an individual patch token, a cell (a $k\times k$ grouping of neighboring patches), a subsample (a non-local grouping of distant patches), a scale (coarse-to-fine resolution), or even a whole image. Additionally, we reformulate discrete token classification as continuous entity regression, leveraging flow-matching methods at each AR step. This approach conditions training on noisy entities instead of ground truth tokens, leading to Noisy Context Learning, which effectively alleviates exposure bias. As a result, xAR offers two key advantages: (1) it enables flexible prediction units that capture different contextual granularity and spatial structures, and (2) it mitigates exposure bias by avoiding reliance on teacher forcing. On ImageNet-256 generation benchmark, our base model, xAR-B (172M), outperforms DiT-XL/SiT-XL (675M) while achieving 20$\times$ faster inference. Meanwhile, xAR-H sets a new state-of-the-art with an FID of 1.24, running 2.2$\times$ faster than the previous best-performing model without relying on vision foundation modules (e.g., DINOv2) or advanced guidance interval sampling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。