用不可优化的约束构建可验证搜索任务,破解大模型评估难题。
Breaking the Evaluation Paradox: Evaluating High-Entropy Search with Computationally Irreducible Constraints

- 设计不可优化的计算约束,强制模型完整遍历搜索空间。
- 生成无限数量带精确真值的测试用例,难度可控且成本极低。
- 适合评估和训练需要系统探索能力的复杂智能体。
评估大语言模型的穷尽搜索能力面临根本性悖论:验证完整性需要完整真值,但高熵枚举任务使人类无法生成真值,导致基准系统性惩罚超越人工标注者的模型。尽管网络搜索与深度研究代理已能发起数百次查询、跨多元站点并生成长篇报告,评估仍依赖部分标注答案集、基于LLM的评判者或单答案问题,回避真正穷尽的搜索场景。本文通过将评估范式从模拟混乱现实转向构建计算纯净挑战,提出VERITAS(可验证遍历搜索评估)框架。该框架基于计算不可约约束原则,引入新型非可优化约束,创建可验证、稀疏答案的搜索任务,其计算等价于穷尽枚举。这些约束易于验证却无法被模型或搜索引擎优化,迫使智能体真实遍历整个搜索空间。VERITAS可自动生成近乎无限的测试用例,具备完美真值与精准难度控制,实例成本主要由哈希计算主导。这不仅为不确定性下的系统性探索提供稳健基准,也为提升此类关键但未充分发展的能力提供了可扩展的训练数据生成方法。
原文摘要 · Abstract (English)
Evaluating the exhaustive search capabilities of large language models (LLMs) is plagued by a fundamental paradox: verifying completeness requires complete ground truth, yet high-entropy enumeration tasks make such ground truth impossible for humans to create. This causes benchmarks to systematically penalize models for outperforming their human annotators. Despite rapid progress in web-search and deep research agents -- which now issue hundreds of queries, traverse diverse sites, and synthesize long reports -- evaluation still largely relies on partially annotated answer sets, LLM-based judges, or single-answer questions that avoid genuinely exhaustive search scenarios. We break this paradox by shifting the evaluation paradigm from simulating a messy reality to constructing computationally pure challenges. We introduce VERITAS (Verifiable Traversal Assessment for Search), a framework built on the principle of computationally irreducible constraints. By introducing novel, non-optimizable constraints, we create verifiable, sparse-answer search tasks that are computationally equivalent to exhaustive enumeration. These constraints are easy to verify but impossible for LLMs or search engines to optimize, forcing agents to genuinely traverse the entire search space. VERITAS can automatically generate a virtually infinite number of test cases with perfect ground truth and precise difficulty control, with marginal instance cost dominated by hash computations. This provides not only a robust benchmark for evaluating systematic exploration under uncertainty but also a scalable method for generating training data to improve these crucial, yet underdeveloped, capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。