测试大模型对残障人士的偏见，发现回应更负面且易出错。
AccessEval: Benchmarking Disability Bias in Large Language Models
- 用真实场景中带残障描述和中性描述的提问对比测试模型
- 残障相关提问下模型输出更负面、刻板且错误率更高
- 适合关注算法公平性与残障权益的研究者和开发者
大型语言模型（LLMs）在多个领域广泛应用，但常在处理真实问题时表现出差异。为系统评估不同残障情境下的影响，我们提出「AccessEval（无障碍评估）」基准，测试21个闭源与开源LLM在6个真实领域、9类残障类型上的表现。采用配对的中性与残障感知查询，通过情感、社会认知与事实准确性指标评估输出。分析显示，残障感知提问的回复普遍更负面、刻板化程度更高，且事实错误率上升。这些偏差随领域与残障类型显著变化，听力、言语及行动障碍受影响尤为严重。结果揭示模型行为中嵌入的持续性能力主义倾向。通过在真实决策场景中评估模型表现，更清晰展现偏见如何转化为对残障用户的实际伤害。该框架弥合了技术评估与用户影响之间的差距，凸显日常应用中偏见缓解的重要性。数据集已公开于：https://huggingface.co/datasets/Srikant86/AccessEval
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly deployed across diverse domains but often exhibit disparities in how they handle real-life queries. To systematically investigate these effects within various disability contexts, we introduce \textbf{AccessEval (Accessibility Evaluation)}, a benchmark evaluating 21 closed- and open-source LLMs across 6 real-world domains and 9 disability types using paired Neutral and Disability-Aware Queries. We evaluated model outputs with metrics for sentiment, social perception, and factual accuracy. Our analysis reveals that responses to disability-aware queries tend to have a more negative tone, increased stereotyping, and higher factual error compared to neutral queries. These effects show notable variation by domain and disability type, with disabilities affecting hearing, speech, and mobility disproportionately impacted. These disparities reflect persistent forms of ableism embedded in model behavior. By examining model performance in real-world decision-making contexts, we better illuminate how such biases can translate into tangible harms for disabled users. This framing helps bridges the gap between technical evaluation and user impact, reinforcing importance of bias mitigation in day-to-day applications. Our dataset is publicly available at: https://huggingface.co/datasets/Srikant86/AccessEval
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。