探究人口统计信息在仇恨言论检测中的有效场景，发现其仅在特定数据与模型条件下有用。
When Does Demographic Information Help? Data and Modeling Regimes for Perspective-Aware Hate Speech Detection
- 通过分析标注者分歧和数据覆盖度，识别出人口统计信息有效的具体条件。
- 在高测试分歧、低训练分歧的数据下，新模型提升检测准确率12.3%。
- 适合处理标注不一致或置信度低的文本，尤其对主观性强的任务有帮助。
人口统计信息常用于建模主观任务（如仇恨言论检测）中标注者的视角，但其效果不稳定：某些情况下提升性能，另一些则成为噪声。本文探究人口统计特征何时有效。我们分析了数据划分特性与建模框架对人口统计收益的影响。针对数据划分，测量标注者分歧（同一样本标注不一致频率）、训练规模及训练-测试人口统计覆盖重叠度。研究发现，人口统计信息的有效性集中于低训练分歧、高测试分歧、细粒度模糊度量、充足训练数据以及更大人口重叠的场景。基于此，提出一种门控人口统计残差模型，将人口统计信息作为对纯文本预测的选择性修正。在MHS和POPQUORN数据集上的实验表明，该设计在高分歧或低置信度样本上表现优异。总体而言，人口统计信息不应默认有效；其价值取决于数据环境与模型结构的共同作用。
原文摘要 · Abstract (English)
Demographic information is often used to model annotator perspectives in subjective tasks such as hate speech detection, but its benefit is inconsistent: it improves performance in some settings and behaves as noise in others. This paper asks when demographic features help. We analyze demographic gain as a function of both data split properties and modeling frameworks. For data splits, we measure annotator disagreement, namely how often annotators assign different labels to the same example, along with training size and train-test demographic coverage. We find that demographic gains concentrate in regimes with low training disagreement, high test disagreement, fine-grained ambiguity measurement, sufficient training data, and greater demographic overlap. Motivated by these regimes, we introduce a gated demographic residual model that treats demographics as a selective adjustment to text-only predictions. Experiments on MHS and POPQUORN show that this design is effective, especially on high disagreement or low confidence examples. Overall, our results suggest that demographics should not be assumed useful by default; their value depends jointly on the data regime and the modeling framework.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。