首个专为扩散型多模态大模型设计的可视化解释方法
Diffusion-CAM: Faithful Visual Explanations for dMLLMs
- 通过可微探针捕捉Transformer中的中间表征,生成类激活图
- 在定位准确性和视觉保真度上超越现有最佳方法
- 适合研究多模态生成模型可解释性的研究人员
尽管扩散型多模态大语言模型(dMLLMs)在多模态生成任务中取得显著进展,但其可解释性机制的发展滞后于架构演进。与产生序列化激活的传统自回归模型不同,扩散架构通过并行去噪生成令牌,导致整个序列呈现平滑、分布式的激活模式。因此,针对局部序列依赖设计的现有类激活映射(CAM)方法不适用于此类非自回归行为。为此,我们提出Diffusion-CAM,首个专为dMLLMs设计的可解释性方法。通过可微探针分析Transformer主干网络中的中间表示,生成原始激活图,以同时捕捉潜在特征及其类别特异性梯度。为解决原始信号固有的随机性,引入四个关键模块,以消除空间模糊性,缓解图像内混杂因素及冗余令牌相关性。大量实验表明,Diffusion-CAM在定位精度和视觉保真度方面显著优于当前最优方法,确立了理解扩散多模态系统并行生成过程的新标准。
原文摘要 · Abstract (English)
While diffusion Multimodal Large Language Models (dMLLMs) have recently achieved remarkable strides in multimodal generation, the development of interpretability mechanisms has lagged behind their architectural evolution. Unlike traditional autoregressive models that produce sequential activations, diffusion-based architectures generate tokens via parallel denoising, resulting in smooth, distributed activation patterns across the entire sequence. Consequently, existing Class Activation Mapping (CAM) methods, which are tailored for local, sequential dependencies, are ill-suited for interpreting these non-autoregressive behaviors. To bridge this gap, we propose Diffusion-CAM, the first interpretability method specifically tailored for dMLLMs. We derive raw activation maps by differentiably probing intermediate representations in the transformer backbone, accordingly capturing both latent features and their class-specific gradients. To address the inherent stochasticity of these raw signals, we incorporate four key modules to resolve spatial ambiguity and mitigate intra-image confounders and redundant token correlations. Extensive experiments demonstrate that Diffusion-CAM significantly outperforms SoTA methods in both localization accuracy and visual fidelity, establishing a new standard for understanding the parallel generation process of diffusion multimodal systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。