arXiv:2509.15553cs.CVcs.AI2025-09被引 1

利用扩散模型中间特征提升多标签分类效果,性能显著优于传统方法。

Diffusion-Based Cross-Modal Feature Extraction for Multi-Label Classification

  • 从扩散Transformer模型中提取图像与文本的中间特征,选择最优时间步与层融合。
  • 在MS-COCO-enhanced上达98.6% mAP,Visual Genome 500上达45.7% mAP,超越主流模型。
  • 发现'Layer 12'在图像任务中表现最佳,提出高效搜索算法避免盲目尝试。

多标签分类应用广泛,依赖于能捕捉标签间交互的强大表示。本文提出Diff-Feat框架,从预训练的扩散-Transformer模型中提取图像与文本的中间特征,并进行融合以用于下游任务。观察发现:视觉任务中最具判别性的特征出现在扩散过程的中段步骤及Transformer的中段模块;语言任务中最佳特征则出现在无噪声步骤且位于最深层。特别地,在不同数据集上均发现一个神秘现象——对于图像任务(使用DiT-XL/2-256×256),第12层始终表现最佳。为此设计了一种启发式局部搜索算法,快速定位最优的'图像-文本'×'块-时间步'组合,避免全网格搜索。仅通过简单的线性投影与特征相加即可实现当前最优性能:在MS-COCO-enhanced上达到98.6% mAP,Visual Genome 500上达到45.7% mAP,显著优于强基线如CNN、图网络和Transformer。t-SNE与聚类指标显示,Diff-Feat形成的语义簇更紧密,优于单模态方法。代码已开源。

原文摘要 · Abstract (English)

Multi-label classification has broad applications and depends on powerful representations capable of capturing multi-label interactions. We introduce \textit{Diff-Feat}, a simple but powerful framework that extracts intermediate features from pre-trained diffusion-Transformer models for images and text, and fuses them for downstream tasks. We observe that for vision tasks, the most discriminative intermediate feature along the diffusion process occurs at the middle step and is located in the middle block in Transformer. In contrast, for language tasks, the best feature occurs at the noise-free step and is located in the deepest block. In particular, we observe a striking phenomenon across varying datasets: a mysterious "Layer $12$" consistently yields the best performance on various downstream classification tasks for images (under DiT-XL/2-256$\times$256). We devise a heuristic local-search algorithm that pinpoints the locally optimal "image-text"$\times$"block-timestep" pair among a few candidates, avoiding an exhaustive grid search. A simple fusion-linear projection followed by addition-of the selected representations yields state-of-the-art performance: 98.6\% mAP on MS-COCO-enhanced and 45.7\% mAP on Visual Genome 500, surpassing strong CNN, graph, and Transformer baselines by a wide margin. t-SNE and clustering metrics further reveal that \textit{Diff-Feat} forms tighter semantic clusters than unimodal counterparts. The code is available at https://github.com/lt-0123/Diff-Feat.

多标签分类扩散模型特征提取Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。