arXiv:2512.16636cs.CV2025-12被引 4

用全局与局部语义联合建模,提升扩散模型生成质量与训练速度

REGLUE Your Latents with Global and Local Semantics for Entangled Diffusion

  • 将图像潜空间、局部视觉语义和全局图像标记统一在同一个框架中建模
  • 在ImageNet上相比基线模型提升FID并加速收敛,关键依赖非线性语义压缩
  • 适合关注图像生成质量与训练效率的扩散模型研究者

潜空间扩散模型(LDMs)虽在图像生成上达到顶尖水平,但其重建式去噪目标仅提供间接语义监督,导致高层语义缓慢涌现,需更长训练时间且限制生成质量。现有方法或通过外部对齐注入视觉基础模型(VFMs)语义,或仅在扩散过程中联合建模有限的VFM特征,未能充分利用丰富的多层空间语义。本文提出REGLUE(Representation Entanglement with Global-Local Unified Encoding),在一个SiT主干网络中联合建模:(i) VAE图像潜空间,(ii) 紧凑的局部(块级)VFM语义,(iii) 全局(图像级)[CLS]标记。一个轻量级卷积语义压缩器非线性地聚合多层VFM特征,生成低维、空间结构化的表示,并与VAE潜变量在扩散过程中纠缠。外部对齐损失进一步引导内部表示向冻结的VFM目标对齐。在ImageNet 256x256上,REGLUE持续优于SiT-B/2和SiT-XL/2基线,以及REPA、ReDi和REG。大量实验表明:(a) 空间语义至关重要,(b) 非线性压缩是释放其全部潜力的关键,(c) 全局标记与外部对齐作为互补的轻量增强,在全局-局部-潜变量联合建模框架中表现良好。

原文摘要 · Abstract (English)

Latent diffusion models (LDMs) achieve state-of-the-art image synthesis, yet their reconstruction-style denoising objective provides only indirect semantic supervision: high-level semantics emerge slowly, requiring longer training and limiting sample quality. Recent works inject semantics from Vision Foundation Models (VFMs) either externally via representation alignment or internally by jointly modeling only a narrow slice of VFM features inside the diffusion process, under-utilizing the rich, nonlinear, multi-layer spatial semantics available. We introduce REGLUE (Representation Entanglement with Global-Local Unified Encoding), a unified latent diffusion framework that jointly models (i) VAE image latents, (ii) compact local (patch-level) VFM semantics, and (iii) a global (image-level) [CLS] token within a single SiT backbone. A lightweight convolutional semantic compressor nonlinearly aggregates multi-layer VFM features into a low-dimensional, spatially structured representation, which is entangled with the VAE latents in the diffusion process. An external alignment loss further regularizes internal representations toward frozen VFM targets. On ImageNet 256x256, REGLUE consistently improves FID and accelerates convergence over SiT-B/2 and SiT-XL/2 baselines, as well as over REPA, ReDi, and REG. Extensive experiments show that (a) spatial VFM semantics are crucial, (b) non-linear compression is key to unlocking their full benefit, and (c) global tokens and external alignment act as complementary, lightweight enhancements within our global-local-latent joint modeling framework. The code is available at https://github.com/giorgospets/reglue .

扩散模型图像生成语义融合潜空间建模

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。