提出一种无需重训练的统一去偏方法，有效降低视觉语言模型的性别偏见。
A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks
- 通过特征裁剪与低置信度补全结合实现跨模态去偏
- 在零样本分类等任务中显著降低性别偏见且不损失性能
- 适用于多种任务，无需重新训练，适合实际部署
视觉语言模型（VLMs）在处理文本和图像数据的同时，已推动人工智能在复杂多模态任务中的发展。然而，这些模型常表现出社会刻板印象引发的偏见，亟需去偏策略。现有方法局限于特定模态或任务，且需大量重训练。本文提出选择性特征插补去偏（SFID），融合特征裁剪与低置信度插补（LCI），有效减少VLMs偏见。该方法通用性强，保持输出语义完整性，且无需重训练，成本低。实验表明，SFID在零样本分类、文本到图像检索、图像描述生成和文本到图像生成等多种任务中，显著降低性别偏见，同时维持模型性能。该方法不仅提升VLM应用公平性，也保障其在多样化场景下的效率与实用性。
原文摘要 · Abstract (English)
Recent advancements in Vision-Language Models (VLMs) have enabled complex multimodal tasks by processing text and image data simultaneously, significantly enhancing the field of artificial intelligence. However, these models often exhibit biases that can skew outputs towards societal stereotypes, thus necessitating debiasing strategies. Existing debiasing methods focus narrowly on specific modalities or tasks, and require extensive retraining. To address these limitations, this paper introduces Selective Feature Imputation for Debiasing (SFID), a novel methodology that integrates feature pruning and low confidence imputation (LCI) to effectively reduce biases in VLMs. SFID is versatile, maintaining the semantic integrity of outputs and costly effective by eliminating the need for retraining. Our experimental results demonstrate SFID's effectiveness across various VLMs tasks including zero-shot classification, text-to-image retrieval, image captioning, and text-to-image generation, by significantly reducing gender biases without compromising performance. This approach not only enhances the fairness of VLMs applications but also preserves their efficiency and utility across diverse scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。