提出HInter方法，自动检测大模型中的多重属性歧视
HInter: Exposing Hidden Intersectional Bias in Large Language Models
- 通过变异分析与依赖解析生成测试用例
- 14.61%测试用例暴露交叉偏见，16.62%为隐藏错误
- 可有效降低误报率，适合模型安全评估者使用
大型语言模型可能对具有多重属性（如种族与性别）的个体表现出歧视，但发现此类交叉偏见极具挑战性。为此，我们提出HInter，一种结合变异分析、依赖解析和元编程不变量的测试技术，可自动检测大模型中的交叉偏见。HInter通过系统性地对句子进行多重变异生成测试输入，利用依赖关系不变性验证输入有效性，并通过对比原始句与变异句的模型输出来检测偏见。我们在六种模型架构和18个模型（包括GPT3.5、Llama2、BERT等）上评估HInter，发现14.61%的生成输入揭示了交叉偏见。结果还表明，依赖不变量使误报率降低一个数量级。此外，观察到16.62%的交叉偏见错误是隐藏的，即其对应的单一属性情况并不触发偏见。该研究强调了对大模型进行交叉偏见测试的重要性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) may portray discrimination towards certain individuals, especially those characterized by multiple attributes (aka intersectional bias). Discovering intersectional bias in LLMs is challenging, as it involves complex inputs on multiple attributes (e.g. race and gender). To address this challenge, we propose HInter, a test technique that synergistically combines mutation analysis, dependency parsing and metamorphic oracles to automatically detect intersectional bias in LLMs. HInter generates test inputs by systematically mutating sentences using multiple mutations, validates inputs via a dependency invariant and detects biases by checking the LLM response on the original and mutated sentences. We evaluate HInter using six LLM architectures and 18 LLM models (GPT3.5, Llama2, BERT, etc) and find that 14.61% of the inputs generated by HInter expose intersectional bias. Results also show that our dependency invariant reduces false positives (incorrect test inputs) by an order of magnitude. Finally, we observed that 16.62% of intersectional bias errors are hidden, meaning that their corresponding atomic cases do not trigger biases. Overall, this work emphasize the importance of testing LLMs for intersectional bias.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。