arXiv:2603.22125cs.CV2026-03被引 3

通过细节对齐提升扩散模型压缩效率,实现更高分辨率生成。

DA-VAE: Plug-in Latent Compression for Diffusion via Detail Alignment

  • 用新增通道编码高分辨率细节,保持原潜空间结构。
  • 仅需轻量微调即可实现1024×1024图像生成,令牌数减少4倍。
  • 适合追求高效高分辨率生成的开发者与研究者。

降低令牌数量对高分辨率潜在扩散模型的训练与推理效率至关重要。现有方法常采用高压缩率图像令牌化器,但仅依赖重建训练会导致高维潜在空间失去有意义结构,增加扩散训练难度。现有方案引入语义对齐或选择性丢弃等额外目标,但通常需代价高昂的扩散模型重训练。而预训练扩散模型本身已具备结构化、低维的潜在空间,因此我们提出细节对齐变分自编码器(DA-VAE),在不改变预训练扩散主干的前提下,仅通过轻量级适配即可提升潜在压缩率。DA-VAE采用显式潜在布局:前C个通道直接来自基础分辨率的预训练VAE,额外D个通道用于编码高分辨率细节。通过简单细节对齐机制,确保扩展后的潜在空间保留原始结构。结合暖启动微调策略,该方法使Stable Diffusion 3.5仅用32×32令牌即可生成1024×1024图像,令牌数为原模型的1/4,在5个H100天内完成。同时支持2048×2048图像生成,速度提升6倍,且图像质量无损。我们在ImageNet上定量验证了方法及设计选择的有效性。

原文摘要 · Abstract (English)

Reducing token count is crucial for efficient training and inference of latent diffusion models, especially at high resolution. A common strategy is to build high-compression image tokenizers with more channels per token. However, when trained only for reconstruction, high-dimensional latent spaces often lose meaningful structure, making diffusion training harder. Existing methods address this with extra objectives such as semantic alignment or selective dropout, but usually require costly diffusion retraining. Pretrained diffusion models, however, already exhibit a structured, lower-dimensional latent space; thus, a simpler idea is to expand the latent dimensionality while preserving this structure. We therefore propose \textbf{D}etail-\textbf{A}ligned VAE, which increases the compression ratio of a pretrained VAE with only lightweight adaptation of the pretrained diffusion backbone. DA-VAE uses an explicit latent layout: the first $C$ channels come directly from the pretrained VAE at a base resolution, while an additional $D$ channels encode higher-resolution details. A simple detail-alignment mechanism encourages the expanded latent space to retain the structure of the original one. With a warm-start fine-tuning strategy, our method enables $1024 \times 1024$ image generation with Stable Diffusion 3.5 using only $32 \times 32$ tokens, $4\times$ fewer than the original model, within 5 H100-days. It further unlocks $2048 \times 2048$ generation with SD3.5, achieving a $6\times$ speedup while preserving image quality. We also validate the method and its design choices quantitatively on ImageNet.

扩散模型潜空间压缩细节对齐高效生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。