发现少数视觉特征决定大模型对人的社会偏见。
StylisticBias: A Few Human Visual Cues Drive Most Social Biases in MLLMs

- 固定身份只变一个视觉属性,精准测量偏见来源。
- 15个特征贡献近80%判断差异,集中在风格与体型。
- 适合研究多模态模型偏见的学者与安全评估者。
多模态大语言模型在个人与社会关键场景中日益普及,但影响其人物评判的视觉线索仍不明确。以往研究常混淆外貌与身份差异。我们提出StylisticBias,一个受控基准,用于评估多模态大模型中的属性级社会偏见。生成500张逼真基础人脸,每张生成约50个单属性变异,共约2.5万张图像。该设计保持身份不变,仅改变一个视觉属性,从而可量化特定线索对模型判断的影响。我们在6个MLLM上评估了25种二元社会判断任务。结果表明,年龄和体型主导身份级效应,而时尚风格等视觉线索引发最大属性级变化。约15个属性解释了近80%的总变化,显示偏见集中于少数视觉线索。敏感性在与外貌语义相关的判断中最强,尤其是经济地位与风格相关判断。我们公开StylisticBias作为细粒度偏见评估基准。代码与数据集:https://github.com/timo-cavelius/StylisticBias 及 https://hf.co/datasets/shaghayegh/stylistic-bias-dataset。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) are increasingly deployed in personally and societally consequential settings, yet the visual cues that shape how these models judge people remain poorly understood. Prior work often compares different (groups of) individuals, making it difficult to separate appearance effects from identity differences. We introduce StylisticBias, a controlled benchmark for evaluating attribute-level social bias in MLLMs. We generate 500 photorealistic base faces and create about 50 single-attribute variations per face, producing about 25K images. This design keeps identity fixed and changes one visual attribute at a time. It lets us measure how specific cues shift model judgments. We evaluate six MLLMs across 25 binary social judgment scenarios. We find that age and body type dominate identity-level effects, while fashion style and other visual cues drive the largest attribute-level shifts. We further find that about 15 attributes account for nearly 80\% of the total variation, showing that bias is concentrated in a small set of visual cues. Sensitivity is strongest in judgments that are semantically aligned with appearance, especially socioeconomic and style-related judgments. We release StylisticBias as a benchmark for fine-grained bias evaluation in multimodal models. Code and dataset: https://github.com/timo-cavelius/StylisticBias and https://hf.co/datasets/shaghayegh/stylistic-bias-dataset.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。