让大模型标注更公平，通过多视角分析提升主观任务准确性
Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks
- 以群体标注分布为关注点，而非单一真实答案
- 在少量人工标注下显著提升难建模群体的预测准确率
- 适合需要公平性评估的文本评价任务
大语言模型用于文本标注时，其输出反映某些人类视角优于其他视角。现有纠错方法假设存在单一真实标签，但在主观任务中，不同人群间的分歧本身具有意义。本文提出视角驱动推理（Perspective-Driven Inference），将群体标注分布作为核心关注量，仅用少量人工标注预算即可估计。我们设计了自适应采样策略，将人工标注资源集中于大模型表现最差的人群。在礼貌性和冒犯性评分任务上的实验表明，相比均匀采样基线，该方法在更难建模的人群中实现了针对性提升，同时保持全面覆盖。
原文摘要 · Abstract (English)
Large language models are increasingly used to annotate texts, but their outputs reflect some human perspectives better than others. Existing methods for correcting LLM annotation error assume a single ground truth. However, this assumption fails in subjective tasks where disagreement across demographic groups is meaningful. Here we introduce Perspective-Driven Inference, a method that treats the distribution of annotations across groups as the quantity of interest, and estimates it using a small human annotation budget. We contribute an adaptive sampling strategy that concentrates human annotation effort on groups where LLM proxies are least accurate. We evaluate on politeness and offensiveness rating tasks, showing targeted improvements for harder-to-model demographic groups relative to uniform sampling baselines, while maintaining coverage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。