提出语义仿射一致性,提升扩散模型的图像生成质量。
AffineTok: Semantic Affine Consistency for Diffusion-Friendly Visual Tokenizer

- 通过语义仿射一致性优化潜空间语义组织,使去噪更准确。
- 在ImageNet上实现1.21(无引导)和1.10(有引导)的gFID新纪录。
- 适合关注扩散模型潜空间设计与生成质量优化的研究者。
视觉分词器越来越多地将语义监督注入潜空间以简化下游扩散过程。然而,如何组织这些语义以促进去噪仍缺乏研究。本文定义了语义恢复目标:去噪过程应从噪声潜空间中恢复干净图像的语义内容,且优秀分词器应使其更易实现。现有方法通过投影器直接从噪声潜空间预测语义,但实际应匹配平均干净潜空间的语义。更重要的是,我们证明语义恢复误差可正交分解为直接预测误差与两者间的偏差。因此,我们识别出一致性作为缺失要求,称之为语义仿射一致性(SAC)。为验证其与生成质量的关系,引入M_SAC作为分词器侧代理指标,在评估分词器与扩散模型规模下,与SiT-XL gFID相关性达0.960,支持以SAC指导训练。进而提出AffineTok,通过两个仅训练阶段组件实现SAC:全局语义协调分词(GSCT)保持干净潜空间语义平均的意义;后验均值语义对齐(PMSA)从噪声输入预测后验均值潜空间并监督其语义。在ImageNet 256上,相比基线,AffineTok在20轮时降低gFID 26%,持续训练后达到无引导1.21、有引导1.10的当前最优gFID。
原文摘要 · Abstract (English)
Visual tokenizers increasingly inject semantic supervision into latent spaces to make downstream diffusion easier. Yet how these semantics should be organized to facilitate denoising remains underexplored. In this paper, we define the semantic recovery objective: the denoising process should recover the semantic content of the clean image from noisy latent, and a good tokenizer should make it easier. Existing approaches train a projector to predict the semantics directly from the noisy latent. We argue that this predicts the average of clean-image semantics, whereas what really needs to be aligned is the semantics of averaged clean latents. More importantly, we demonstrate that the semantic recovery error orthogonally decomposes into the error of the optimal semantic prediction directly from the noisy latent and the error between these two predictions. We therefore identify their consistency as the missing requirement and call it Semantic Affine Consistency (SAC). To examine whether this overlooked requirement is closely related to downstream generation, we introduce M_SAC, a tokenizer-side proxy for SAC. Across the evaluated tokenizers and diffusion model scales, M_SAC closely tracks generation quality, reaching a Pearson correlation of 0.960 with SiT-XL gFID, thereby motivating SAC-guided tokenizer training. We then introduce AffineTok, which promotes SAC through two complementary, training-only components. Global Semantic Coordination Token (GSCT) coordinates the semantic organization of clean latents, keeping semantic averaging meaningful, while Posterior-Mean Semantic Alignment (PMSA) predicts posterior-mean latents from noisy inputs and supervises their semantics. On ImageNet 256, compared with the baseline, AffineTok reduces gFID by 26% at 20 epochs and, with continued training, achieves a new state-of-the-art gFID of 1.21 without classifier-free guidance and 1.10 with guidance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。