揭示扩散分类器的决策偏差,发现其依赖背景而非主体。
How Do Diffusion Classifiers Decide? A Bias-Centric Evaluation

- 通过三个维度测试扩散分类器的决策偏差:属性绑定、大小顺序偏好、背景依赖。
- 在ComCo上对大小顺序的依赖比OpenCLIP高,在ImageNet-B上背景依赖导致准确率大幅下降。
- 热图和注意力可视化揭示了决策机制,对生成模型也有警示意义。
扩散模型被用于零样本分类,形成通过最小化噪声预测误差来匹配最佳文本提示的扩散分类器。尽管应用日益广泛,其决策机制仍不清晰。本文提出ASOB-Bench,从属性绑定、大小顺序偏差、背景依赖三个维度评估扩散分类器。在新构建的数据集上扩展五个新属性类别,发现扩散分类器在属性误绑定方面优于OpenCLIP基线;在现有ComCo基准上对大小顺序更敏感;在ImageNet-B上准确率显著下降,显示对背景线索的强依赖。重建误差热图与U-Net交叉注意力可视化揭示了各类偏差的内在机制。由于扩散分类器与文生图模型共用同一去噪器,这些单次诊断也指向生成中的类似失败模式。总体而言,扩散分类器表现出与判别式视觉语言模型不同的偏差特征,为构建更鲁棒的扩散模型提供指导。
原文摘要 · Abstract (English)
Diffusion models have recently been repurposed for zero-shot classification, giving rise to diffusion classifiers that identify the best-matching text prompt by minimizing the noise-prediction error. Despite their growing adoption, how these models make classification decisions remains poorly understood. We introduce ASOB-Bench, a bias evaluation for diffusion classifiers along three dimensions: Attribute binding, Size-Order bias, and Background dependency. These dimensions serve not as an exhaustive taxonomy but as targeted probes of how the text-conditioned reconstruction-error score reaches a decision. Such a perspective is well studied for discriminative vision-language models, yet remains overlooked for diffusion classifiers. Extending an existing framework with five new attribute categories on newly constructed datasets, we find diffusion classifiers are less prone to attribute misbinding than an OpenCLIP baseline; on the established ComCo benchmark they are substantially more susceptible to size-order shortcuts; and on ImageNet-B they suffer far larger accuracy drops, revealing heavy reliance on background over foreground cues. Reconstruction-error heatmaps and U-Net cross-attention visualizations expose the mechanism behind each bias. Because diffusion classifiers share the same denoiser as text-to-image models, these single-pass diagnostics also point toward analogous failure modes in generation. Overall, diffusion classifiers exhibit a distinct bias profile from vision-language models, offering guidance for building more robust diffusion-based models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。