评测智能体在需设置检索状态的复杂网页任务中的表现，发现多数系统因状态配置错误失败。
SGR-Bench: Benchmarking Search Agents on State-Gated Retrieval
- 构建包含100个专家任务的SGR-Bench基准，模拟真实网站的状态依赖检索场景。
- 最强系统仅达66.18%项级F1，行级F1更低，主要败在状态配置错误而非答案生成。
- 关键问题为检索范围漂移和筛选条件不匹配，适合研究智能体推理与工具调用能力者参考。
大型语言模型与工具型智能体的发展拓展了网络任务的评测范畴，但一类重要的专业检索任务仍缺乏充分刻画。在许多专业数据检索网站中，答案相关的证据仅在通过过滤器、视图、层级或作用域正确设置检索状态后才可获取。我们称此能力为状态门控检索（SGR）。本文提出SGR-Bench，一个针对该场景的基准测试，包含100个专家设计的任务，覆盖六个数据源家族和12个公共数据生态。每个任务要求识别正确的网站并配置其特有的检索状态以生成结构化答案。SGR-Bench同时提供约束引导与目标导向两种问题形式，支持对显式与隐式引导在状态门控检索中效果的可控比较。我们评估了八个基于CLI的智能体大模型系统及三个商业搜索代理产品。在SGR-Bench上，最强系统仅达到66.18%的项级F1，行级F1显著更低。对156条可分析的失败命令行轨迹的手动审计显示：智能体常能定位到相关网站，但未能建立正确的站点特定检索状态。检索范围漂移（37.2%）和标准不匹配（27.6%）是主要失败原因，而最终答案组合错误占比仅10.3%。数据集与单例评估说明已公开于https://huggingface.co/datasets/PKUAIWeb/SGR-BENCH。
原文摘要 · Abstract (English)
Recent advances in large language models and tool-using agents have expanded the range of benchmarked web tasks. Yet an important class of specialized retrieval tasks remains undercharacterized. On many specialized data-retrieval websites, answer-bearing evidence becomes accessible only after establishing the correct site-specific retrieval state through filters, views, hierarchies, or scopes. We term this capability state-gated retrieval (SGR). We introduce SGR-Bench, a benchmark for this setting containing 100 expert-curated tasks spanning six source families and 12 public data ecosystems. Each task requires discovering the appropriate website and configuring its site-specific retrieval state to produce a structured answer. SGR-Bench pairs constraint-guided and goal-oriented formulations of the same underlying problems, enabling controlled comparisons between explicit and implicit guidance for state-gated retrieval. We evaluate eight CLI-based agentic LLM systems and three commercial search-agent products. On SGR-Bench, the strongest system reaches only 66.18% item-level F1, while row-level F1 remains much lower. A manual audit of 156 analyzable failed CLI trajectories shows why: agents often reach a relevant web source, but establish the wrong site-specific retrieval state. Retrieval-scope drift (37.2%) and criterion mismatch (27.6%) dominate, whereas final answer composition accounts for only 10.3%. The dataset and single-case evaluation instructions are available at https://huggingface.co/datasets/PKUAIWeb/SGR-BENCH.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。