构建3000万图像的评测基准，揭示视觉语言模型隐含的社会偏见。
VIGNETTE: Socially Grounded Bias Evaluation for Vision-Language Models
- 基于社会心理学设计四维度测评框架，覆盖事实、感知、刻板印象与决策
- 在3000万+图像上发现模型对身份特征的隐蔽刻板联想与歧视模式
- 适合关注AI伦理、公平性研究者及模型评估团队使用
尽管大型语言模型的偏见问题已受到广泛研究，但视觉语言模型（VLMs）的相关研究仍相对不足。现有研究多聚焦于肖像类图像和性别-职业关联，忽视了更广泛、更复杂的社会刻板印象及其潜在危害。本文提出VIGNETTE，一个包含3000万+图像的大规模视觉问答（VQA）基准，通过涵盖事实性、感知、刻板印象与决策四个维度的问答框架，评估VLM中的偏见。超越传统局限，该研究考察模型在具体语境中如何解读身份，揭示其在特质与能力判断上的假设及歧视模式。结合社会心理学理论，分析模型如何将视觉身份线索与特质及角色推断相联系，从而编码社会等级结构。研究发现了一系列微妙、多维且出人意料的刻板模式，揭示了VLM如何从输入中构建社会意义。
原文摘要 · Abstract (English)
While bias in large language models (LLMs) is well-studied, similar concerns in vision-language models (VLMs) have received comparatively less attention. Existing VLM bias studies often focus on portrait-style images and gender-occupation associations, overlooking broader and more complex social stereotypes and their implied harm. This work introduces VIGNETTE, a large-scale VQA benchmark with 30M+ images for evaluating bias in VLMs through a question-answering framework spanning four directions: factuality, perception, stereotyping, and decision making. Beyond narrowly-centered studies, we assess how VLMs interpret identities in contextualized settings, revealing how models make trait and capability assumptions and exhibit patterns of discrimination. Drawing from social psychology, we examine how VLMs connect visual identity cues to trait and role-based inferences, encoding social hierarchies, through biased selections. Our findings uncover subtle, multifaceted, and surprising stereotypical patterns, offering insights into how VLMs construct social meaning from inputs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。