通过关系视角衡量大模型间偏见相似性，揭示公平性本质是相对而非孤立的。
Bias Similarity Measurement: A Black-Box Audit of Fairness Across LLMs
- 将多种偏见信号融合为统一相似性空间，实现跨模型公平性对比
- 30个模型测试显示小模型在强制选择下公平性反而下降，指令微调主要导致回避回答
- 适用于采购审计、版本回滚与模型溯源，支持多语言和代码场景
大型语言模型（LLMs）复现社会偏见，但现有评估常孤立评价单个模型，掩盖了偏见在模型家族和版本间的延续性。本文提出偏见相似性测量（BSM），将公平性视为模型间的相对属性，整合标量、分布、行为与表征信号至统一相似性空间。对超过100万条提示语评估30个大模型后发现：指令微调主要促使模型回避回答而非改变内部表征；小模型在强制选择下准确率提升有限且公平性可能恶化；开源模型可达到甚至超越专有系统表现。不同模型家族呈现显著差异：Gemma倾向拒绝，LLaMA 3.1更趋中立且拒绝较少，整体趋向于高回避行为。反直觉的是，Gemma 3 Instruct以远低于GPT-4的成本达到其公平性水平，而Gemini的过度回避抑制了实用性。除上述发现外，BSM还提供采购审计、回归测试与来源追溯的工作流，并自然扩展至代码与多语言场景。结果表明，公平性不应被看作孤立评分，而应作为比较性的偏见相似性，从而实现对大模型生态系统的系统性审计。代码已公开于https://github.com/HyejunJeong/bias_llm。
原文摘要 · Abstract (English)
Large Language Models (LLMs) reproduce social biases, yet prevailing evaluations score models in isolation, obscuring how biases persist across families and releases. We introduce Bias Similarity Measurement (BSM), which treats fairness as a relational property between models, unifying scalar, distributional, behavioral, and representational signals into a single similarity space. Evaluating 30 LLMs on 1M+ prompts, we find that instruction tuning primarily enforces abstention rather than altering internal representations; small models gain little accuracy and can become less fair under forced choice; and open-weight models can match or exceed proprietary systems. Family signatures diverge: Gemma favors refusal, LLaMA 3.1 approaches neutrality with fewer refusals, and converges toward abstention-heavy behavior overall. Counterintuitively, Gemma 3 Instruct matches GPT-4-level fairness at far lower cost, whereas Gemini's heavy abstention suppresses utility. Beyond these findings, BSM offers an auditing workflow for procurement, regression testing, and lineage screening, and extends naturally to code and multilingual settings. Our results reframe fairness not as isolated scores but as comparative bias similarity, enabling systematic auditing of LLM ecosystems. Code available at https://github.com/HyejunJeong/bias_llm.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。