HART用混合分块方式高效生成1024×1024高清图像，质量媲美扩散模型。
HART: Efficient Visual Generation with Hybrid Autoregressive Transformer
- 混合分块编码：离散+连续双通道，分别处理整体结构与细节残差。
- 重建FID降至0.30，生成FID达5.38，优于现有自回归模型31%。
- 速度比扩散模型快7.7倍，显存占用低13.4倍，适合部署场景。
我们提出混合自回归变压器（HART），一种可直接生成1024×1024图像的自回归视觉生成模型，图像质量可与扩散模型媲美。现有自回归模型受限于离散分词器的重建质量差以及生成1024像素图像时训练成本过高。为此，我们设计了混合分词器，将自编码器产生的连续潜变量分解为两部分：表示整体构图的离散令牌和表示无法由离散令牌表达的残差成分的连续令牌。离散部分由可扩展分辨率的离散自回归模型建模，连续部分则通过仅含3700万参数的轻量级残差扩散模块学习。相比纯离散VAR分词器，我们的方法在MJHQ-30K数据集上重建FID从2.11提升至0.30，生成FID从7.85降低至5.38，改善31%。HART在FID和CLIP得分上均超越当前最优扩散模型，吞吐量高出4.5–7.7倍，乘加操作数（MACs）减少6.9–13.4倍。代码已开源：https://github.com/mit-han-lab/hart。
原文摘要 · Abstract (English)
We introduce Hybrid Autoregressive Transformer (HART), an autoregressive (AR) visual generation model capable of directly generating 1024x1024 images, rivaling diffusion models in image generation quality. Existing AR models face limitations due to the poor image reconstruction quality of their discrete tokenizers and the prohibitive training costs associated with generating 1024px images. To address these challenges, we present the hybrid tokenizer, which decomposes the continuous latents from the autoencoder into two components: discrete tokens representing the big picture and continuous tokens representing the residual components that cannot be represented by the discrete tokens. The discrete component is modeled by a scalable-resolution discrete AR model, while the continuous component is learned with a lightweight residual diffusion module with only 37M parameters. Compared with the discrete-only VAR tokenizer, our hybrid approach improves reconstruction FID from 2.11 to 0.30 on MJHQ-30K, leading to a 31% generation FID improvement from 7.85 to 5.38. HART also outperforms state-of-the-art diffusion models in both FID and CLIP score, with 4.5-7.7x higher throughput and 6.9-13.4x lower MACs. Our code is open sourced at https://github.com/mit-han-lab/hart.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。