用隐式平衡层让图像生成模型更省显存、可调计算量,效果还更好。
Visual Implicit Autoregressive Modeling

- 在浅层块间加入隐式平衡层,用无雅可比反向传播训练
- 参数减少61.6%,FID达2.16,峰值显存降至8.53GB
- 支持按需调节每阶段计算量,适合部署在资源受限设备
基于下一尺度预测的视觉自回归模型(VAR)虽生成质量高,但深层堆叠导致每尺度计算量固定且高分辨率下显存膨胀。本文提出视觉隐式自回归建模(VIAR),在浅层前后块间嵌入隐式平衡层,使用无雅可比反向传播训练,实现恒定训练显存;推理时引入每尺度迭代控制旋钮,可灵活调节计算量。在ImageNet 256x256基准上,VIAR仅用VAR 38.4%的参数即达到FID 2.16、sFID 8.07,性能匹配或超越强自回归基线,并保持与大型扩散模型相当的竞争力。通过调节旋钮,峰值显存从19.24GB降至8.53GB,单张RTX 4090吞吐量从15.16提升至32.08张/秒,无需重训练。消融实验表明,较少步数即可收敛,且在所有质量-效率权衡点上持续优于VAR。零样本修复与类别条件编辑任务中,VIAR生成细节更锐利、边界更平滑,同时保持全局结构,验证了隐式平衡与每尺度计算控制的实际价值。
原文摘要 · Abstract (English)
Visual Autoregressive Modeling (VAR) based on next-scale prediction achieves strong generation quality, but their explicit deep stacks fix the amount of computation per scale and inflate memory at high resolutions. We introduce Visual Implicit Autoregressive Modeling (VIAR), a next-scale autoregressive generator that embeds an implicit equilibrium layer between shallow pre/post blocks. The implicit layer is trained with Jacobian-Free Backpropagation, yielding constant training memory, while inference exposes a per-scale iteration knob that enables compute control. On ImageNet 256x256 benchmark, VIAR attains FID 2.16, and sFID 8.07 with only 38.4% parameters of VAR, matching or surpassing strong AR baselines and remaining competitive with large diffusion models. By controlling the per-scale knob, VIAR can reduce peak memory from 19.24 GB to 8.53 GB and doubles throughput from 15.16 to 32.08 images/s on a single RTX 4090, without retraining. Ablations show that fewer steps are sufficient for fixed-point iterations to converge and that VIAR consistently dominates VAR across quality efficiency operating points. In zero shot in-painting and class-conditional editing, VIAR produces sharper details and smoother boundaries while preserving global structure, validating the benefits of implicit equilibria and per-scale compute control for practical, deployable visual generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。