针对大模型对残障群体的隐性伤害，构建了由残障者共创的评估框架。
DisaBench: A Participatory Evaluation Framework for Disability Harms in Language Models
- 基于12类残障伤害分类，设计正向与对抗性提示配对评测方法
- 175个提示覆盖7个生活领域，525组响应中残障类型影响伤害率差异
- 强调残障伤害具情境性，需领域专家识别细微风险，适合安全评估开发者
通用大模型安全基准未能充分评估残障相关伤害。我们提出DisaBench：一个由残障人士与红队专家共同构建的十二类残障伤害分类体系，一种基于分类的评测方法，通过在七个生活领域中配对良性与对抗性提示，并构建包含175个提示的数据集，对525组提示-响应对进行人工标注。四位具有残障经历的评估者发现：伤害率随残障类型显著变化，且在非文本模态中会加剧；术语相关伤害具有文化和时间依赖性，无法普适判断；标准安全评估仅能捕捉明显错误，而难以发现仅靠领域专长才能识别的细微伤害。残障伤害兼具个人性、交叉性与社区定义性，无法脱离个体全貌独立评估。我们将在Hugging Face上开源数据集、分类体系与方法，提供无需额外基础设施的开源红队框架，可直接集成至现有安全流程。
原文摘要 · Abstract (English)
General-purpose safety benchmarks for large language models do not adequately evaluate disability-related harms. We introduce DisaBench: a taxonomy of twelve disability harm categories co-created with people with disabilities and red teaming experts, a taxonomy-driven evaluation methodology that pairs benign and adversarial prompts across seven life domains, and a dataset of 175 prompts with human-annotated labels on 525 prompt-response pairs. Annotation by four evaluators with lived disability experience reveals three findings: harm rates vary sharply by disability type and will compound in non-text modalities, terminology-driven harm is culturally and temporally bound rather than universally assessable, and standard safety evaluation catches overt failures while missing the subtle harms that only domain expertise can recognize. Disability harm is simultaneously personal, intersectional, and community-defined: it cannot be isolated from the full context of who a person is, and general-purpose benchmarks systematically miss it. We will release the dataset, taxonomy, and methodology via Hugging Face and an open-source red teaming framework for direct integration into existing safety pipelines with no additional infrastructure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。