无需训练，通过锚点传播实现多模态扩散模型精准定位
AnchorDiff: Training-Free Concept Grounding for MM-DiTs via Anchor-Based Graph Propagation
- 用高置信锚点作为种子，在混合图上进行语义传播
- 在混淆数据集上概念泄漏降低67%，定位更准确
- 适合需要零样本定位的视觉语言任务开发者
多模态扩散变换器（MM-DiTs）具备零样本概念定位能力，但现有基于注意力的方法常在视觉相似概念间产生重叠激活，导致目标响应溢出至非目标物体，这种现象称为概念泄漏。为此，我们提出AnchorDiff，一种无需训练的定位方法，将语义定位与结构优化解耦。该方法从概念到图像的注意力图中选取高置信锚点，并将其作为一热种子，在由图像自注意力构建的混合图上进行传播。该图利用输出空间相似性实现物体内部密集传播，并通过行级注意力门控抑制跨物体连接。此外，我们构建了多概念混淆数据集（Multi-Concept Confusion Dataset），包含具有多个视觉相似概念的图像及独立掩码，可显式评估概念泄漏。实验表明，AnchorDiff在ImageNet-Segmentation和PascalVOC上表现优异，且在多概念混淆数据集上概念泄漏显著减少。
原文摘要 · Abstract (English)
Multi-Modal Diffusion Transformers (MM-DiTs) encode rich representations for training-free concept grounding, but existing attention-based methods often produce overlapping activations on visually confusable concepts, a failure mode we call concept leakage, where target responses spill over to non-target objects. To address this issue, we propose AnchorDiff, a training-free grounding method that decouples semantic localization from structural refinement. AnchorDiff selects a high-confidence anchor from concept-to-image attention map and propagates it as a one-hot seed over a hybrid graph derived from image-to-image self-attention. The graph uses output-space similarity for dense within-object propagation and a row-wise attention gate to suppress cross-object connections. Additionally, we introduce the Multi-Concept Confusion Dataset, which contains images with multiple visually similar concepts and separate masks, enabling explicit evaluation of concept leakage. Experiments show that AnchorDiff achieves strong grounding performance on ImageNet-Segmentation and PascalVOC, while substantially reducing concept leakage on our Multi-Concept Confusion Dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。