通过显式建模数据流形几何,提升扩散模型生成质量与效率
Diffusion Image Generation with Explicit Modeling of Data Manifold Geometry

- 将离散补丁令牌融入连续扩散得分函数,融合离散结构与连续生成优势
- 训练80轮后无引导FID达22.73,较基线降低15.95(对比DiT)和9.06(对比SiT)
- 小参数量模型即超越大模型,适合追求高效高质生成的开发者
图像生成模型旨在从底层数据流形中采样数据点,这需要学习并解码一个密集、低维且紧凑的参数化空间。为此,我们提出数据流形感知图像扩散模型(MIND),通过将离散补丁标记化集成到连续扩散模型的得分函数中,显式建模流形几何。该方法有效结合了离散标记的结构量化能力与连续扩散的并行生成灵活性。此外,我们通过新颖的软top-k聚合机制实现端到端可微训练,并引入双分支高频特征嵌入层以缓解Transformer主干对低维输入的谱偏差。针对推理,设计了多阶段过渡采样方案,根据时间步动态调整采样策略。在ImageNet 256×256上的大量实验表明,经过80轮训练,基础模型在无引导条件下达到22.73的FID,几乎将原生DiT-B/2基线的43.47降至一半。相比基线DiT和SiT,FID平均分别降低15.95和9.06。在带引导生成上,仅含130M参数的MIND-B模型取得2.06的FID,超越参数量达31亿的LlamaGen-3B;MIND-XL(715M参数)进一步将FID降至1.95。本工作为基于扩散的图像生成提供了新视角,推动该领域未来研究。
原文摘要 · Abstract (English)
Image generative models aim to sample data points from the underlying data manifold, a task that requires learning and decoding a dense, low-dimensional, and compact parameterization space. To achieve this, we propose the Data Manifold-aware Image diffusioN moDel (MIND), a novel framework that explicitly models manifold geometry by integrating discrete patch tokenization into the score function of a continuous diffusion model. This approach successfully leverages both the structural quantification capabilities of discrete tokens and the parallel generation flexibility of continuous diffusion. Moreover, we enable end-to-end differentiable training via a novel soft top-$k$ aggregation mechanism and introduce dual-branch high-frequency feature embedding layers to alleviate the spectral bias of transformer backbones on low-dimensional inputs. Furthermore, for inference, we design a multi-stage transition sampling scheme that dynamically adjusts the sampling scheme based on timestep. Extensive experiments on ImageNet 256$\times$256 demonstrate the effectiveness of MIND. After 80-epoch training, our base model achieves an FID of 22.73 without guidance, nearly halving the 43.47 FID of the vanilla DiT-B/2 baseline. The proposed method reduces FID by 15.95 and 9.06 on average compared with the baselines DiT and SiT, respectively. For image generation on ImageNet-256$\times$256 with guidance, the proposed MIND-B with only 130M parameters achieves an FID of 2.06, superpassing the LlamaGen-3B with 3.1B parameters. The proposed MIND-XL with 715M parameters further reduces the FID to 1.95. Our MIND introduces a fresh perspective on diffusion-based image generation, paving the way for future research and innovation in this community. The code will be publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。