评测大模型提取细粒度证据的能力，发现小模型反而更接近人类标注。
Can LLMs extract human-like fine-grained evidence for evidence-based fact-checking?
- 用人工标注的捷克语和斯洛伐克语评论证据数据集，评估大模型抽取精确文本片段的能力。
- 小模型Llama3.1:8b正确率高，而参数量更大的GPT-OSS-120b表现不佳。
- 通义千问、DeepSeek等中等规模模型在准确性和与人标注一致上表现平衡。
在线新闻评论中的虚假信息频繁传播，亟需有效方法检测事实性错误。为有力支持或反驳评论中提取的论断，需识别相关文档并精确定位能证明或反驳该论断的文本片段。本文聚焦于后一任务——对捷克语和斯洛伐克语论断进行细粒度证据抽取。我们构建了新的数据集，包含由付费标注者进行双向标注的细粒度证据。在该数据集上评估大型语言模型（LLMs）的表现，以衡量其与人类标注的一致性。结果表明，大模型常无法从源文本中逐字复制证据，导致输出无效。误差分析显示，尽管参数量较小，Llama3.1:8b 模型仍实现较高比例的正确输出；而参数量更大的 GPT-OSS-120b 表现欠佳。此外，Qwen3:14b、DeepSeek-R1:32b 和 GPT-OSS:20b 模型在模型规模与人类标注一致性之间展现出良好平衡。
原文摘要 · Abstract (English)
Misinformation frequently spreads in user comments under online news articles, highlighting the need for effective methods to detect factually incorrect information. To strongly support or refute claims extracted from such comments, it is necessary to identify relevant documents and pinpoint the exact text spans that justify or contradict each claim. This paper focuses on the latter task -- fine-grained evidence extraction for Czech and Slovak claims. We create new dataset, containing two-way annotated fine-grained evidence created by paid annotators. We evaluate large language models (LLMs) on this dataset to assess their alignment with human annotations. The results reveal that LLMs often fail to copy evidence verbatim from the source text, leading to invalid outputs. Error-rate analysis shows that the {llama3.1:8b model achieves a high proportion of correct outputs despite its relatively small size, while the gpt-oss-120b model underperforms despite having many more parameters. Furthermore, the models qwen3:14b, deepseek-r1:32b, and gpt-oss:20b demonstrate an effective balance between model size and alignment with human annotations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。