快速精准评估大模型长文本事实性,效率超现有方法。
FaStfact: Faster, Stronger Long-Form Factuality Evaluations in LLMs
- 分块提取主张并预验证,降低计算开销
- 基于网页证据检索,准确率优于基线
- 适合需要高效可靠评估的AI研究者
评估大语言模型生成长文本的事实性仍面临效率与可靠性挑战。现有方法通过分解文本为主张、搜寻证据并验证,但存在流程复杂导致效率低,以及主张提取不准、证据不足影响效果的问题。为此,我们提出FaStfact评估框架,在保持人类评估高对齐度的同时,实现现有基线中最佳的时间与令牌效率。该框架采用基于置信度的分块主张提取与预验证机制,显著减少耗时与资源消耗;在证据获取阶段,从网络爬取文档级证据,并在验证时按需检索。基于自建标注基准数据集FaStfact-Bench的大量实验表明,FaStfact能高效且准确地评估长文本事实性。代码、数据及标注工具已开源。
原文摘要 · Abstract (English)
Evaluating the factuality of long-form generations from Large Language Models (LLMs) remains challenging due to efficiency bottlenecks and reliability concerns. Prior efforts attempt this by decomposing text into claims, searching for evidence, and verifying claims, but suffer from critical drawbacks: (1) inefficiency due to overcomplicated pipeline components, and (2) ineffectiveness stemming from inaccurate claim sets and insufficient evidence. To address these limitations, we propose \textbf{FaStfact}, an evaluation framework that achieves the highest alignment with human evaluation and time/token efficiency among existing baselines. FaStfact first employs chunk-level claim extraction integrated with confidence-based pre-verification, significantly reducing the time and token cost while ensuring reliability. For searching and verification, it collects document-level evidence from crawled web-pages and selectively retrieves it during verification. Extensive experiments based on an annotated benchmark \textbf{FaStfact-Bench} demonstrate the reliability of FaStfact in both efficiently and effectively evaluating long-form factuality. Code, benchmark data, and annotation interface tool are available at https://github.com/Yingjia-Wan/FaStfact.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。