提出无需训练的去偏框架DebFilter,减轻文本到图像生成中的性别年龄偏见。
DebFilter: Eradicating Biases Stashed in Value

- 通过调整跨注意力中值向量的固定偏移,实现推理时去偏。
- 在多个基准测试中显著降低生成图像的性别与年龄偏见指数。
- 轻量级无训练设计,适合作为现有模型的即插即用去偏模块。
文本到图像扩散模型通过多步去噪过程生成图像,其指导信号来自预训练视觉-语言模型(如CLIP)提取的文本嵌入。然而,这些嵌入本身蕴含性别、年龄等社会与语义偏见,经由引导机制及大规模数据集的不平衡训练被放大,导致生成结果出现偏差。本文提出DebFilter,一种轻量级且无需训练的去偏框架。观察到每一步去噪过程中模型误差主要受跨注意力动态影响,我们引入一种针对跨注意力值组件的偏置修正策略:对引导嵌入的特定切片施加固定偏移,有效将跨注意力值的语义方向导向无偏表示。该调整重构了得分空间,使输出更平衡,同时保持与原始文本语义的一致性。相比依赖微调或重训练的方法,DebFilter完全在推理阶段运行,无需额外数据或模型更新。实验表明,该方法能有效缓解生成图像中的社会偏见,为更公平、包容的文本到图像生成提供高效可扩展的路径。
原文摘要 · Abstract (English)
Text-to-image diffusion models, which are theoretically equivalent to score-based generative models, generate images through a multi-step denoising process guided by text embeddings extracted from pretrained vision-language models such as CLIP. However, these text embeddings inherently encode social and semantic biases -- such as those related to gender and age -- that are subsequently propagated and amplified through the guidance mechanism, along with the model's training on large-scale datasets that are imbalanced with respect to these bias-related concepts, often leading to skewed outputs in text-to-image generation. We propose DebFilter, a lightweight and training-free framework for mitigating such biases in text-to-image diffusion models. Observing that the model's error prediction at each denoising step is primarily influenced by cross-attention dynamics, we introduce a bias-correction strategy that adjusts the value components within cross-attention. Specifically, we apply a fixed offset to the slice of guidance embedding, effectively steering the semantic direction of cross-attention values toward unbiased representations. This adjustment reconfigures the score landscape to produce balanced outputs while maintaining alignment with the intended text semantics. Unlike prior approaches that rely on fine-tuning or retraining, DebFilter operates entirely at inference time, requiring no additional data or model updates. Our results demonstrate that this method effectively mitigates social biases in generated images, offering an efficient and scalable pathway toward fairer and more inclusive text-to-image generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。