arXiv:2601.06931cs.CVcs.AI2026-01ACL被引 2

用真人面部反事实图像,精准测量视觉语言模型中的社会偏见。

Measuring Social Bias in Vision-Language Models with Face-Only Counterfactuals from Real Photos

  • 仅修改人脸特征,保留其他视觉元素,实现种族与性别影响的分离评估。
  • 在6个职业、10个人口群体中构建480张场景匹配的反事实图像数据集。
  • 发现不同任务设计下偏见表现差异大,提示需重视评估方法的设计。

视觉语言模型在社会关键场景中应用日益广泛,引发对其由人口统计特征引发的社会偏见的担忧。衡量此类偏见的核心挑战在于视觉混杂下的归因问题:真实图像中种族、性别与背景、服饰等因子纠缠,难以分离。本文提出「仅面部反事实评估范式」,通过仅编辑与种族、性别相关的面部属性,保持其余视觉因素不变,从真实照片生成反事实样本。基于此,构建了包含480张场景匹配反事实图像的FOCUS数据集(覆盖6个职业、10个人口群体),并提出包含三项决策导向任务的REFLECT基准:二选一判断、多选社会经济推断、数值薪资建议。对五种主流VLM的实验表明,在严格视觉控制下,人口统计差异仍显著存在,且结果随任务设计变化剧烈。研究强调了受控反事实审计的必要性,并指出任务设计是评估多模态模型社会偏见的关键因素。

原文摘要 · Abstract (English)

Vision-Language Models (VLMs) are increasingly deployed in socially consequential settings, raising concerns about social bias driven by demographic cues. A central challenge in measuring such social bias is attribution under visual confounding: real-world images entangle race and gender with correlated factors such as background and clothing, obscuring attribution. We propose a \textbf{face-only counterfactual evaluation paradigm} that isolates demographic effects while preserving real-image realism. Starting from real photographs, we generate counterfactual variants by editing only facial attributes related to race and gender, keeping all other visual factors fixed. Based on this paradigm, we construct \textbf{FOCUS}, a dataset of 480 scene-matched counterfactual images across six occupations and ten demographic groups, and propose \textbf{REFLECT}, a benchmark comprising three decision-oriented tasks: two-alternative forced choice, multiple-choice socioeconomic inference, and numeric salary recommendation. Experiments on five state-of-the-art VLMs reveal that demographic disparities persist under strict visual control and vary substantially across task formulations. These findings underscore the necessity of controlled, counterfactual audits and highlight task design as a critical factor in evaluating social bias in multimodal models.

社会偏见视觉语言模型反事实评估公平性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。