用搜索增强评估框架，让大模型自检答案真实性。
SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA
- 构建可检索外部信息的智能代理，动态生成查询并整合证据
- 在多个问答基准上与人类评价达成高度一致，最高接近完美
- 适合评估大模型事实性，尤其对训练外问题更可靠
随着大语言模型（LLMs）在开放域问答中的广泛应用，依赖静态预标注参考答案的评估方式面临成本高、可扩展性差和覆盖不全的问题。而仅用模型自身作为评判者又因缺乏外部验证，常过度接受错误答案、编造支持理由，且在训练数据之外的问题上表现显著下降。本文提出搜索增强评估（SAGE）框架，无需固定真实答案即可评估大模型输出。SAGE作为智能代理，通过迭代生成网络查询、收集外部信息、总结发现并基于反思优化后续搜索，实现动态证据融合。相比传统依赖静态参考或纯模型自评的方法，SAGE降低了对固定参考的依赖，提供了一种可扩展、自适应的事实性评估方案。在多个自由形式问答基准上的实验表明，SAGE与人类评估达成显著至近乎完美的一致性。
原文摘要 · Abstract (English)
As Large Language Models (LLMs) become increasingly used for question-answering (QA), relying on static, pre-annotated references for evaluation poses significant challenges in cost, scalability, and completeness. Meanwhile, using LLMs themselves as evaluators without external grounding remains unreliable for objective tasks, as they systematically over-accept incorrect answers, fabricate supporting rationales, and degrade sharply on questions that fall outside their training data. We propose Search-AuGmented Evaluation (SAGE), a framework to assess LLM outputs without fixed ground-truth answers. Unlike conventional metrics that compare to static references or depend solely on LLM-as-a-judge knowledge, SAGE acts as an agent that actively retrieves and synthesizes external evidence. It iteratively generates web queries, collects information, summarizes findings, and refines subsequent searches through reflection. By reducing dependence on static reference-driven evaluation protocols, SAGE offers a scalable and adaptive alternative for evaluating the factuality of LLMs. Experimental results on multiple free-form QA benchmarks show that SAGE achieves substantial to perfect agreement with human evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。