构建政治表态不一致检测基准，助力识别虚假承诺。
Misleading through Inconsistency: A Benchmark for Political Inconsistencies Detection
- 提出不一致检测任务并定义多层级类型体系。
- 构建698对标注语句数据集，237条附推理说明。
- 大模型表现接近人类，但细粒度分类仍有提升空间。
政治表态中的不一致属于一种误导性信息，会削弱公众信任并阻碍问责。自动检测不一致可帮助记者提出质询，增强政治透明度。本文提出不一致检测任务，并建立不一致类型分级体系以推动该方向的自然语言处理研究。为支持政治领域不一致检测，我们构建了一个包含698对人工标注的政治陈述的数据集，其中237个样本附有标注者推理说明。语料主要来自德国Wahl-O-Mat和瑞士Smartvote等投票辅助平台，反映真实政治议题。我们在该数据集上对大型语言模型（LLMs）进行基准测试，结果表明，总体上它们在检测不一致性方面表现与人类相当，甚至可能优于单个个体人类，在预测群体标注真值方面更具优势。然而，在识别细粒度不一致类型时，现有模型尚未达到性能上限（受标注自然差异影响），仍具优化空间。数据集与代码已公开可用。
原文摘要 · Abstract (English)
Inconsistent political statements represent a form of misinformation. They erode public trust and pose challenges to accountability, when left unnoticed. Detecting inconsistencies automatically could support journalists in asking clarification questions, thereby helping to keep politicians accountable. We propose the Inconsistency detection task and develop a scale of inconsistency types to prompt NLP-research in this direction. To provide a resource for detecting inconsistencies in a political domain, we present a dataset of 698 human-annotated pairs of political statements with explanations of the annotators' reasoning for 237 samples. The statements mainly come from voting assistant platforms such as Wahl-O-Mat in Germany and Smartvote in Switzerland, reflecting real-world political issues. We benchmark Large Language Models (LLMs) on our dataset and show that in general, they are as good as humans at detecting inconsistencies, and might be even better than individual humans at predicting the crowd-annotated ground-truth. However, when it comes to identifying fine-grained inconsistency types, none of the model have reached the upper bound of performance (due to natural labeling variation), thus leaving room for improvement. We make our dataset and code publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。