用大模型自动生成情感分析偏见测试用例，省去人工标注成本。
BTC-SAM: Leveraging LLMs for Generation of Bias Test Cases for Sentiment Analysis Models
- 基于大模型可控生成多样化的偏见测试句。
- 生成句子在语言多样性上优于传统提示方法，覆盖更广。
- 适合需快速评估模型社会偏见的研究者与开发者。
情感分析（SA）模型存在固有社会偏见，可能在实际应用中造成伤害。这些偏见可通过分析仅在主体身份群体上变化的句子输出来识别。构建自然、语言丰富、相关且多样化的句子集以充分覆盖领域，成本高昂，尤其涉及广泛偏见时，需要领域专家或众包支持。本文提出新型偏见测试框架BTC-SAM，利用大语言模型（LLMs）以最少配置实现对情感分析模型偏见测试用例的高质量生成。实验表明，依赖大模型可生成高语言变体和多样性，即使面对未见过的偏见，测试覆盖率也优于基础提示方法。
原文摘要 · Abstract (English)
Sentiment Analysis (SA) models harbor inherent social biases that can be harmful in real-world applications. These biases are identified by examining the output of SA models for sentences that only vary in the identity groups of the subjects. Constructing natural, linguistically rich, relevant, and diverse sets of sentences that provide sufficient coverage over the domain is expensive, especially when addressing a wide range of biases: it requires domain experts and/or crowd-sourcing. In this paper, we present a novel bias testing framework, BTC-SAM, which generates high-quality test cases for bias testing in SA models with minimal specification using Large Language Models (LLMs) for the controllable generation of test sentences. Our experiments show that relying on LLMs can provide high linguistic variation and diversity in the test sentences, thereby offering better test coverage compared to base prompting methods even for previously unseen biases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。