arXiv:2607.24063cs.AI2026-07

提出新评估协议,让大模型事实性与算力成本公开比拼。

The Cost of Knowing: A Resource-Aware Protocol for Benchmarking Hallucination Beyond Static Leaderboards

论文配图:The Cost of Knowing: A Resource-Aware Protocol for Benchmarking Hallucination Beyond Static Leaderboards
图 1 · 摘自论文原文
  • 用资源感知的对抗机制替代静态评分,量化回答成本
  • 实测显示:高准确率模型可能因耗能过高而排名垫底
  • 适合关注部署效率与真实性能平衡的研究者

在标准事实性任务上,前沿模型已接近性能上限。问题正从‘系统是否准确’转向‘准确需付出多少计算成本’。静态排行榜仅孤立评估准确率,视计算为免费,无法区分真正优的系统与单纯烧资源的系统。例如,一个暴力型Best-of-4代理虽有更高原始得分(H-Score 0.9169 vs 0.9103),但其消耗约四倍的令牌与延迟,计入成本后反而在Q-Score(0.5169 vs 0.5217)中落后。为此,我们提出MAS-HQ(多智能体系统幻觉挑战)——一种资源感知的评估协议,可封装任意事实性检测器,并对系统进行竞争性对比而非孤立打分。Q-Score衡量的是事实性减去归一化成本。在摘要与开放域问答任务中,单智能体基线趋于资源密集型过度优化,而竞争机制则引导出更高效的策略。这些改进虽小但稳定,且在100次试验中保持一致。该框架对前沿模型(Gemini-2.5-Pro、GPT-5)仍具判别力,其原始得分已接近天花板。

原文摘要 · Abstract (English)

On standard factuality tasks, frontier models now cluster near the top of the scale. The question is therefore shifting from how factual a system is toward how much compute that factuality costs. Static leaderboards score factuality in isolation and treat compute as free, so they cannot tell a genuinely better system apart from one that simply spends more. Consider a ranking reversal. A brute-force Best-of-4 agent posts the higher raw factuality score (H-Score 0.9169 vs 0.9103) and would top a static leaderboard, but once cost is counted it is the worse system, losing on Q-Score (0.5169 vs 0.5217) at roughly four times the tokens and latency, under a reported cost weight whose sensitivity we sweep. So the system that tops a static leaderboard can be the worse one to deploy. To make this trade-off visible, we introduce MAS-HQ (Multi-Agent System Hallucination Quest), a resource-aware evaluation protocol. It wraps any factuality detector and normalizes for cost, and it pits systems against each other rather than scoring them in isolation. The Q-Score measures factuality minus normalized cost under a competitive match. Across summarization and open-domain QA, single-agent baselines drift into resource-heavy over-optimization, while competition elicits more resource-efficient policies. These gains are small but consistent, and stable across 100 trials. The axis stays discriminative for frontier systems (Gemini-2.5-Pro, and GPT-5) whose raw factuality scores are already bunched near the ceiling. MAS-HQ provides a reproducible way to measure how much a factual answer costs.

评估协议资源效率幻觉检测模型竞赛

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。