专家与大模型协作定义性别歧视，提升零样本检测效果。
Tell Me What You Know About Sexism: Expert-LLM Interaction Strategies and Co-Created Definitions for Zero-Shot Sexism Detection
- 专家与GPT3.5互动共创性别歧视定义，激发更复杂表述。
- 大模型生成的定义平均优于专家独立撰写，部分合作者表现更佳。
- 适合对性别歧视检测、人机协同研究感兴趣的学者参考。
本文研究性别歧视研究者与大型语言模型（LLM）之间的混合智能协作，采用四阶段流程：首先，九位性别歧视研究者回答关于性别歧视及大模型认知的问题；随后参与两项与GPT3.5的交互实验——第一项评估模型对性别歧视的认知水平及其研究适用性，第二项要求每位专家构建三种定义：专家自写、大模型生成、共同创作。最后，利用每种定义作为提示模板，对GPT4o在五个性别歧视基准数据集上抽取的2,500条文本进行零样本分类，共产生67,500次分类决策。结果表明，人机协作促使定义更长、更复杂；大模型生成的定义平均表现优于专家独立撰写的版本；但部分专家（包括不熟悉大模型者）通过合作定义显著提升了分类性能。
原文摘要 · Abstract (English)
This paper investigates hybrid intelligence and collaboration between researchers of sexism and Large Language Models (LLMs), with a four-component pipeline. First, nine sexism researchers answer questions about their knowledge of sexism and of LLMs. They then participate in two interactive experiments involving an LLM (GPT3.5). The first experiment has experts assessing the model's knowledge about sexism and suitability for use in research. The second experiment tasks them with creating three different definitions of sexism: an expert-written definition, an LLM-written one, and a co-created definition. Lastly, zero-shot classification experiments use the three definitions from each expert in a prompt template for sexism detection, evaluating GPT4o on 2.500 texts sampled from five sexism benchmarks. We then analyze the resulting 67.500 classification decisions. The LLM interactions lead to longer and more complex definitions of sexism. Expert-written definitions on average perform poorly compared to LLM-generated definitions. However, some experts do improve classification performance with their co-created definitions of sexism, also experts who are inexperienced in using LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。