用对齐特征实现生成时精准控制,无需修改模型
Test-Time Conditioning with Representation-Aligned Visual Features
- 通过优化特征相似性,在推理时引导生成过程
- 在ImageNet和COCO上实现高质量多样生成
- 支持细粒度纹理到全局语义的多尺度控制
尽管自监督模型的表征对齐已被证明可提升扩散模型训练效果,但其在推理阶段条件控制中的潜力仍不明确。我们提出Representation-Aligned Guidance(REPA-G)框架,利用具有丰富语义属性的对齐表征,在生成过程中实现测试时条件控制。通过在推理阶段优化相似性目标(即势能),我们将去噪过程引导至从预训练特征提取器中提取的条件表征。该方法可在多个尺度实现灵活控制,从单个图像块的精细纹理匹配,到使用全局图像特征令牌的宏观语义引导。进一步扩展至多概念组合,实现不同概念的忠实融合。REPA-G完全在推理阶段运行,提供了一种比模糊文本提示或粗粒度类别标签更灵活、精确的替代方案。我们从理论上证明了该引导机制可使采样服从势能诱导的倾斜分布。在ImageNet和COCO上的定量结果表明,该方法实现了高质量、多样化的生成。代码已开源:https://github.com/valeoai/REPA-G。
原文摘要 · Abstract (English)
While representation alignment with self-supervised models has been shown to improve diffusion model training, its potential for enhancing inference-time conditioning remains largely unexplored. We introduce Representation-Aligned Guidance (REPA-G), a framework that leverages these aligned representations, with rich semantic properties, to enable test-time conditioning from features in generation. By optimizing a similarity objective (the potential) at inference, we steer the denoising process toward a conditioned representation extracted from a pre-trained feature extractor. Our method provides versatile control at multiple scales, ranging from fine-grained texture matching via single patches to broad semantic guidance using global image feature tokens. We further extend this to multi-concept composition, allowing for the faithful combination of distinct concepts. REPA-G operates entirely at inference time, offering a flexible and precise alternative to often ambiguous text prompts or coarse class labels. We theoretically justify how this guidance enables sampling from the potential-induced tilted distribution. Quantitative results on ImageNet and COCO demonstrate that our approach achieves high-quality, diverse generations. Code is available at https://github.com/valeoai/REPA-G.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。