提出新评估框架DoveScore，提升长文本事实一致性检测能力
Long-Form Information Alignment Evaluation Beyond Atomic Facts
- 通过建模事实间关系，联合验证事实准确性和事件顺序
- 在蒙太奇式欺骗数据集上，现有方法AUC低于65%，而新方法提升超8%
- 适合需要高可信度生成的长文本场景，如报告、新闻写作
信息对齐评估在自然语言生成和大模型可信部署中至关重要，有助于减少幻觉并增强用户信任。现有细粒度方法（如FactScore）逐个验证事实，但忽略事实间的依赖关系，易受隐蔽攻击。本文提出MontageLie基准，通过拼接真实语句构造欺骗性叙述，不引入显式幻觉。实验表明，粗粒度与细粒度评估方法均易受此攻击，AUC-ROC低于65%。为此，我们提出DoveScore框架，联合验证事实准确性与事件顺序一致性。通过建模事实间关系，DoveScore相比现有方法性能提升超8%，为长文本对齐评估提供更鲁棒的解决方案。代码与数据集见https://github.com/dannalily/DoveScore。
原文摘要 · Abstract (English)
Information alignment evaluators are vital for various NLG evaluation tasks and trustworthy LLM deployment, reducing hallucinations and enhancing user trust. Current fine-grained methods, like FactScore, verify facts individually but neglect inter-fact dependencies, enabling subtle vulnerabilities. In this work, we introduce MontageLie, a challenging benchmark that constructs deceptive narratives by "montaging" truthful statements without introducing explicit hallucinations. We demonstrate that both coarse-grained LLM-based evaluators and current fine-grained frameworks are susceptible to this attack, with AUC-ROC scores falling below 65%. To enable more robust fine-grained evaluation, we propose DoveScore, a novel framework that jointly verifies factual accuracy and event-order consistency. By modeling inter-fact relationships, DoveScore outperforms existing fine-grained methods by over 8%, providing a more robust solution for long-form text alignment evaluation. Our code and datasets are available at https://github.com/dannalily/DoveScore.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。