让视觉自回归模型生成更符合提示语的图像,解决组合错误问题。
VISTA: Test-Time Compositional Alignment for Visual Autoregressive Generation

- 通过梯度优化中间表示,动态调整生成过程以满足组合约束。
- 在20亿参数模型上提升近20%的组合准确性,空间关系改进最显著。
- 无需训练、不改模型,适合希望提升现有模型表现的研究者。
视觉自回归(VAR)模型已成为文本到图像生成的快速高质量替代方案,但与扩散模型一样,仍存在持续的组合性失败问题,生成图像常违背提示中的属性绑定和空间关系。尽管测试时对齐方法在扩散模型中已很丰富,但针对下一阶段自回归生成的方法尚无,因其状态化、离散化、多分辨率采样过程使现有技术难以适用。本文提出首个基于梯度的测试时对齐框架VISTA(Visual Autoregressive Semantic Test-time Alignment),专为自回归图像生成设计。基于Infinity架构,VISTA在生成过程中直接干预,通过冻结的Transformer优化中间表示,引导视觉预测符合组合约束,无需修改模型参数或额外训练。VISTA引入了跨尺度稳定优化机制,并支持任意可微分约束接入交叉注意力空间。在两个基准上、两种模型规模下,VISTA均提升了所有目标组合类别性能,20亿参数模型平均得分提升近20%,80亿参数模型提升约6%,空间关系收益最大。图像质量保持良好:独立偏好模型评估显示,未优化的VISTA输出得分高出近20%。值得注意的是,20亿参数模型经VISTA后超越四倍大小的基线模型,表明模型规模间的组合差距在测试时可被显著弥补。
原文摘要 · Abstract (English)
Visual autoregressive (VAR) models have emerged as a fast, high-quality alternative to diffusion for text-to-image generation, but like diffusion models they exhibit persistent compositional failures, producing images that violate the attribute bindings and spatial relations specified in the prompt. While a rich line of test-time alignment methods has developed for diffusion, no comparable approach exists for next-scale VAR generation, whose stateful, discrete, multi-resolution sampling process makes existing techniques inapplicable. We close this gap with \textbf{VISTA} (\textbf{Vi}sual Autoregressive \textbf{S}emantic \textbf{T}est-time \textbf{A}lignment), the first gradient-based test-time alignment framework for next-scale autoregressive image generation. Built on Infinity, VISTA intervenes directly in the generation process, optimizing intermediate representations through the frozen transformer to steer visual predictions toward compositional constraints, without modifying model parameters or requiring additional training. VISTA introduces the mechanisms needed to make such optimization stable across scales, together with an extensible objective space that any differentiable constraint on cross-attention can plug into. Across two benchmarks and two model scales, VISTA improves every targeted compositional category, raising the mean targeted score by nearly 20\% on a 2B backbone and almost 6\% on an 8B backbone, with the largest gains on spatial relations. Image quality is preserved: an independent preference model VISTA never optimizes scores its outputs nearly 20\% higher. Notably, the 2B model with VISTA surpasses a backbone four times its size, indicating that a substantial part of the compositional gap between model scales is recoverable at test time.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。