构建首个完整配对的反仇恨言论数据集，提升系统评估可靠性
FC-CONAN: An Exhaustively Paired Dataset for Robust Evaluation of Retrieval Systems
- 穷举45条仇恨言论与129条反叙事的全部组合，生成完整配对数据
- 通过多阶段标注获得四个可靠等级的数据集，发现数百个新正例
- 适用于反仇恨言论检索系统评测与错误分析，开源可用
仇恨言论是在线对话中的关键问题，一种有前景的应对策略是使用反叙事（CNs）。链接仇恨言论与反叙事的数据集对推进反言论研究至关重要。然而，即使如CONAN（Chung等，2019）这样的旗舰资源也仅标注了部分可能的仇恨言论-反叙事配对，限制了评估效果。我们提出FC-CONAN（Fully Connected CONAN），首个通过穷尽考虑45条英文仇恨言论与129条反叙事所有组合而构建的数据集。经过九名标注员和四名验证者参与的两阶段标注流程，生成四个分区——钻石、黄金、白银和青铜——在可靠性与规模间取得平衡。所有标注配对均不与CONAN重叠，揭示了数百个此前未标注的正例。FC-CONAN使反言论检索系统的评估更加真实，并支持细致的错误分析。该数据集已公开。
原文摘要 · Abstract (English)
Hate speech (HS) is a critical issue in online discourse, and one promising strategy to counter it is through the use of counter-narratives (CNs). Datasets linking HS with CNs are essential for advancing counterspeech research. However, even flagship resources like CONAN (Chung et al., 2019) annotate only a sparse subset of all possible HS-CN pairs, limiting evaluation. We introduce FC-CONAN (Fully Connected CONAN), the first dataset created by exhaustively considering all combinations of 45 English HS messages and 129 CNs. A two-stage annotation process involving nine annotators and four validators produces four partitions-Diamond, Gold, Silver, and Bronze-that balance reliability and scale. None of the labeled pairs overlap with CONAN, uncovering hundreds of previously unlabelled positives. FC-CONAN enables more faithful evaluation of counterspeech retrieval systems and facilitates detailed error analysis. The dataset is publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。