让自回归图像生成模型直接优化像素质量,提升生成效果。
VA-$π$: Variational Policy Alignment for Pixel-Aware Autoregressive Generation
- 将生成器与分词器对齐视为变分优化问题,构建像素级目标函数。
- 仅用1%数据和25分钟微调,FID从14.36降至7.65,IS提升至116.70。
- 无需重训练分词器或外部奖励模型,适合快速改进现有生成模型。
自回归视觉生成依赖分词器将图像映射为离散序列,但分词器训练目标是重建干净图像,而自回归生成器仅优化标记似然性,导致生成序列解码后图像质量低下,缺乏像素空间直接监督。为此,本文提出轻量级后训练框架VA-π,直接以像素空间为目标优化自回归模型。VA-π将生成器与分词器对齐建模为变分优化问题,推导出统一像素重建与自回归建模的证据下界(ELBO)。为在离散标记空间中优化,引入基于强化学习的对齐策略,将生成器视为策略,以像素重建质量作为内在奖励。该奖励通过教师强制下的重建效果衡量,使模型获得直接像素级指导,无需昂贵的自由运行采样。ELBO中的正则化项自然保持标记分布一致性。VA-π可快速适配现有自回归生成器,无需分词器重训练或外部奖励模型。仅使用1% ImageNet-1K数据及25分钟调优,即可使LlamaGen-XXL的FID从14.36降至7.65,IS从86.55提升至116.70;在GenEval上,视觉生成模型(LlamaGen)文本到图像任务得分从0.306升至0.339,统一多模态模型(Janus-Pro)从0.725升至0.744。代码已开源。
原文摘要 · Abstract (English)
Autoregressive (AR) visual generation relies on tokenizers to map images to and from discrete sequences. However, tokenizers are trained to reconstruct clean images from ground-truth tokens, while AR generators are optimized only for token likelihood. This misalignment leads to generated token sequences that may decode into low-quality images, without direct supervision from the pixel space. We propose VA-$π$, a lightweight post-training framework that directly optimizes AR models with a principled pixel-space objective. VA-$π$ formulates the generator-tokenizer alignment as a variational optimization, deriving an evidence lower bound (ELBO) that unifies pixel reconstruction and autoregressive modeling. To optimize under the discrete token space, VA-$π$ introduces a reinforcement-based alignment strategy that treats the AR generator as a policy, uses pixel-space reconstruction quality as its intrinsic reward. The reward is measured by how well the predicted token sequences can reconstruct the original image under teacher forcing, giving the model direct pixel-level guidance without expensive free-running sampling. The regularization term of the ELBO serves as a natural regularizer, maintaining distributional consistency of tokens. VA-$π$ enables rapid adaptation of existing AR generators, without neither tokenizer retraining nor external reward models. With only 1% ImageNet-1K data and 25 minutes of tuning, it reduces FID from 14.36 to 7.65 and improves IS from 86.55 to 116.70 on LlamaGen-XXL, while also yielding notable gains in the text-to-image task on GenEval for both visual generation model (LlamaGen: from 0.306 to 0.339) and unified multi-modal model (Janus-Pro: from 0.725 to 0.744). Code is available at https://github.com/Lil-Shake/VA-Pi.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。