用投影器注入特征表示,让扩散模型生成更一致的图像。
Training-Free Representation Guidance for Diffusion Models with a Representation Alignment Projector
- 引入投影器在采样中注入无监督特征表示,作为语义锚点。
- ImageNet上FID降至3.3,比原方法提升显著。
- 无需修改模型结构,适合提升生成一致性与语义保真度。
基于扩散框架的生成模型已实现高质量视觉合成,并支持可控采样和大规模训练。推理阶段的引导方法如无分类器引导和代表性引导通过调整采样动态增强语义对齐,但未能充分利用无监督特征表示。尽管这些视觉表示蕴含丰富语义结构,但在生成过程中因缺乏真实参考图像而受限。本文揭示了扩散变换器在早期去噪阶段存在语义漂移,即使条件相同,随机性也会导致对齐不一致。为此,提出一种基于表示对齐投影器的引导方案,将投影器预测的表示注入中间采样步骤,提供有效语义锚点且不改变模型架构。在SiTs和REPAs上的实验显示,类别条件下的ImageNet合成性能显著提升:例如,REPA-XL/2的FID从5.9降至3.3;该方法在SiT模型上优于代表性引导。与无分类器引导结合时进一步取得互补增益,增强了语义连贯性和视觉保真度。结果表明,基于表示的扩散采样是一种强化语义保持与图像一致性的实用策略。
原文摘要 · Abstract (English)
Recent progress in generative modeling has enabled high-quality visual synthesis with diffusion-based frameworks, supporting controllable sampling and large-scale training. Inference-time guidance methods such as classifier-free and representative guidance enhance semantic alignment by modifying sampling dynamics; however, they do not fully exploit unsupervised feature representations. Although such visual representations contain rich semantic structure, their integration during generation is constrained by the absence of ground-truth reference images at inference. This work reveals semantic drift in the early denoising stages of diffusion transformers, where stochasticity results in inconsistent alignment even under identical conditioning. To mitigate this issue, we introduce a guidance scheme using a representation alignment projector that injects representations predicted by a projector into intermediate sampling steps, providing an effective semantic anchor without modifying the model architecture. Experiments on SiTs and REPAs show notable improvements in class-conditional ImageNet synthesis, achieving substantially lower FID scores; for example, REPA-XL/2 improves from 5.9 to 3.3, and the proposed method outperforms representative guidance when applied to SiT models. The approach further yields complementary gains when combined with classifier-free guidance, demonstrating enhanced semantic coherence and visual fidelity. These results establish representation-informed diffusion sampling as a practical strategy for reinforcing semantic preservation and image consistency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。