用大模型生成虚假关联，让视觉语言模型更公平
PRISM: Reducing Spurious Implicit Biases in Vision-Language Models with LLM-Guided Embedding Projection
- 用大模型生成含虚假关联的场景描述，指导嵌入空间修正
- 在Waterbirds和CelebA上优于现有去偏方法
- 无需额外数据或预定义偏见类别，通用性强
我们提出Projection-based Reduction of Implicit Spurious bias in vision-language Models（PRISM），一种无需数据、任务无关的视觉语言模型（如CLIP）去偏新方法。这类模型常继承并放大训练数据中的偏见，导致预测偏差。PRISM通过两阶段设计实现去偏：首先，利用大模型对简单类别提示生成包含虚假关联的场景描述；其次，采用新型对比式去偏损失，学习一个投影映射，将嵌入映射到降低虚假关联的潜在空间，同时保持图像与文本嵌入的一致性。大量实验表明，PRISM在Waterbirds和CelebA两个常用数据集上均优于现有去偏方法。代码已开源：https://github.com/MahdiyarMM/PRISM。
原文摘要 · Abstract (English)
We introduce Projection-based Reduction of Implicit Spurious bias in vision-language Models (PRISM), a new data-free and task-agnostic solution for bias mitigation in VLMs like CLIP. VLMs often inherit and amplify biases in their training data, leading to skewed predictions. PRISM is designed to debias VLMs without relying on predefined bias categories or additional external data. It operates in two stages: first, an LLM is prompted with simple class prompts to generate scene descriptions that contain spurious correlations. Next, PRISM uses our novel contrastive-style debiasing loss to learn a projection that maps the embeddings onto a latent space that minimizes spurious correlations while preserving the alignment between image and text embeddings.Extensive experiments demonstrate that PRISM outperforms current debiasing methods on the commonly used Waterbirds and CelebA datasets We make our code public at: https://github.com/MahdiyarMM/PRISM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。