评测大模型在长文本中生成事实准确回复的能力。
The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input
- 用长文档+用户请求测试模型是否基于原文回答。
- 响应必须完整符合要求且完全源自输入文档，否则淘汰。
- 多模型评分聚合减少偏差，适合评估严谨性需求场景。
我们提出FACTS Grounding，一个在线排行榜及配套基准，用于评估语言模型在用户提示包含完整文档（最长32k token）时，生成与上下文事实一致的长篇回应能力。每个提示包含用户请求和完整文档，要求模型生成的答案既满足请求又完全基于所提供文档。评估分两阶段进行：(1) 不满足用户请求的响应被直接淘汰；(2) 剩余响应由自动化判别模型判断其是否完全基于文档内容。判别模型通过预留测试集全面筛选最优提示模板，最终的事实性得分由多个判别模型聚合而成，以减轻评估偏差。该排行榜将持续维护，包含公开与私有数据集划分，支持外部参与同时保障公平性。可访问 https://www.kaggle.com/facts-leaderboard 查看。
原文摘要 · Abstract (English)
We introduce FACTS Grounding, an online leaderboard and associated benchmark that evaluates language models' ability to generate text that is factually accurate with respect to given context in the user prompt. In our benchmark, each prompt includes a user request and a full document, with a maximum length of 32k tokens, requiring long-form responses. The long-form responses are required to be fully grounded in the provided context document while fulfilling the user request. Models are evaluated using automated judge models in two phases: (1) responses are disqualified if they do not fulfill the user request; (2) they are judged as accurate if the response is fully grounded in the provided document. The automated judge models were comprehensively evaluated against a held-out test-set to pick the best prompt template, and the final factuality score is an aggregate of multiple judge models to mitigate evaluation bias. The FACTS Grounding leaderboard will be actively maintained over time, and contains both public and private splits to allow for external participation while guarding the integrity of the leaderboard. It can be found at https://www.kaggle.com/facts-leaderboard.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。