开源工具评估大模型生成文本的事实性，支持任意Hugging Face模型。
OpenFActScore: Open-Source Atomic Evaluation of Factuality in Text Generation
- 用原子事实提取与验证机制，逐条检查生成内容真实性。
- 开源模型在关键指标上逼近闭源系统，Gemma表现最佳。
- 适合研究者复现实验或低成本评测大模型事实准确性。
我们提出OpenFActScore，一个开源的FActScore实现框架，用于评估大语言模型生成文本的事实性。FActScore通过原子事实生成（AFG）提取独立事实陈述，并通过原子事实验证（AFV）与可信知识源比对验证。原版FActScore依赖闭源商业模型（如InstructGPT、ChatGPT），而OpenFActScore支持任意Hugging Face兼容模型完成AFG与AFV任务。我们详细说明了实现设计与适配修改，并在原始FActScore基准上评估多个开源模型：AFG使用BERTScore-F1衡量，AFV以相对于人工标注的错误率评估。结果表明，开源模型可近似闭源系统性能，其中Gemma表现最优，最终方案与原始实验呈0.99皮尔逊相关性。该工具提升评估透明度与可复现性，降低使用成本，项目地址：https://github.com/lflage/OpenFActScore。
原文摘要 · Abstract (English)
We introduce OpenFActScore, an open-source implementation of the FActScore framework for evaluating the factuality of text generated by large language models (LLMs). FActScore evaluates the factual accuracy of long-form text by using Atomic Fact Generation (AFG) to extract individual factual claims and Atomic Fact Validation (AFV) to verify each claim against a trusted knowledge source. While the original FActScore relies on closed-source and commercial models such as InstructGPT and ChatGPT, OpenFActScore enables the use of any Hugging Face-compatible model for both AFG and AFV. We provide a detailed technical overview of our implementation, highlighting design choices and modifications made to support open models. We evaluate multiple open-source LLMs on both AFG and AFV using the original FActScore benchmark, reporting BERTScore-F1 for AFG and Error Rate relative to human annotations for AFV. Our results show that open models can approximate the performance of closed-source systems, with Gemma achieving the best overall performance, and our final setup obtains a 0.99 Pearson correlation with the original FActScore experiments. OpenFActScore promotes transparency, reproducibility, and cost-effective evaluation, and is available at: https://github.com/lflage/OpenFActScore.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。