用大模型自动发现并消除视觉识别中的未知偏见
MAVias: Mitigate any Visual Bias
- 通过大模型从自然语言中挖掘潜在视觉偏见
- 在多个数据集上显著降低多种未知偏见影响
- 适合需要高可信度视觉模型的科研与应用
缓解计算机视觉模型中的偏见是提升人工智能可信度的关键。现有方法仅针对预设的少数偏见，难以应对包含多种甚至未知偏见的视觉数据集。为此，我们提出 MAVias，一种开放集偏见缓解方法，利用基础模型发现视觉属性与目标类别间的虚假关联。MAVias 首先通过基础图像标注模型捕捉自然语言中的多样化视觉特征，再利用大语言模型筛选出与目标类别相关的特征，生成一组语言编码的潜在视觉偏见。随后将这些偏见转化为视觉-语言嵌入，并引入一种内处理偏见缓解机制，防止模型学习相关特征。在 CelebA、Waterbirds、ImageNet 和 UrbanCars 等多样数据集上的实验表明，MAVias 能有效检测并缓解广泛存在的偏见，在视觉识别任务中优于当前最先进方法。
原文摘要 · Abstract (English)
Mitigating biases in computer vision models is an essential step towards the trustworthiness of artificial intelligence models. Existing bias mitigation methods focus on a small set of predefined biases, limiting their applicability in visual datasets where multiple, possibly unknown biases exist. To address this limitation, we introduce MAVias, an open-set bias mitigation approach leveraging foundation models to discover spurious associations between visual attributes and target classes. MAVias first captures a wide variety of visual features in natural language via a foundation image tagging model, and then leverages a large language model to select those visual features defining the target class, resulting in a set of language-coded potential visual biases. We then translate this set of potential biases into vision-language embeddings and introduce an in-processing bias mitigation approach to prevent the model from encoding information related to them. Our experiments on diverse datasets, including CelebA, Waterbirds, ImageNet, and UrbanCars, show that MAVias effectively detects and mitigates a wide range of biases in visual recognition tasks outperforming current state-of-the-art.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。