让扩散模型同时生成图像和多种感知信息，提升多模态生成效果。
MMDiff: Extending Diffusion Transformers for Multi-Modal Generation
- 通过轻量解码头融合扩散过程中的多时步特征，实现多模态输出。
- 多时步特征融合使语义分割指标提升28.7% mIoU，显著优于单时步提取。
- 适用于需要高效合成数据或可解释空间引导的多模态任务。
扩散变换器虽具强大生成能力，但其去噪轨迹中丰富的感知表征在内容生成后被丢弃。本文提出MMDiff框架，将冻结的扩散变换器改造为多模态生成系统，可联合生成图像及任意组合的密集感知模态，仅需轻量级解码头。核心发现：感知信息沿去噪轨迹时空分布，采用空间可变聚合权重的多时步特征融合至关重要，使语义分割性能最高提升28.7% mIoU。进一步引入概念驱动注意力提取，实现可解释的空间引导；结果表明，冻结扩散特征在性能上可媲美并互补于DINOv3等先进编码器。仅训练轻量解码头即可在语义分割、显著性目标检测和深度估计任务中取得优异表现，并支持大规模合成数据生成。
原文摘要 · Abstract (English)
Diffusion transformers have demonstrated remarkable generative capabilities, yet the rich perceptual representations computed across their denoising trajectory are discarded once the content is rendered. We present MMDiff, a framework that transforms a frozen diffusion transformer into a multi-modal generative system that jointly produces images alongside any combination of dense perceptual modalities using lightweight decoder heads. Our central finding is that perceptual information is temporally distributed along the denoising trajectory, and that multi-timestep feature fusion with spatially varying aggregation weights is essential, improving semantic segmentation results by up to 28.7% mIoU over single-timestep extraction. We further adopt concept-driven attention extraction for interpretable spatial guidance, and show that frozen diffusion features are competitive with and complementary to state-of-the-art encoders such as DINOv3. By training only lightweight decoder heads on a frozen backbone, we achieve strong performance in semantic segmentation, salient object detection, and depth estimation, and demonstrate that this framework enables effective synthetic data generation at scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。