arXiv:2606.27595cs.CL2026-06

构建首个韩语广度搜索基准,评估网页智能体完整枚举与属性填充能力。

Ko-WideSearch: A Korean Breadth-Search Benchmark for Exhaustive Set Enumeration by Web Agents

  • 基于自动化合成与验证流水线构建韩语广度搜索数据集
  • 228张表覆盖190个实体,跨三层难度,最大成员匹配率达100%
  • 揭示智能体难在找值而非格式化,自由文本字段错误率最高

现有网页智能体评测多聚焦深度搜索——在一系列约束后定位单一隐晦答案;而广度搜索——完整枚举封闭集合并填写每项属性——极少被评估,尤其在非英语语境。广度搜索构建困难:验证黄金标准集的完整性与每个条目正确性成本远高于检查单一答案。本文提出 extsc{Ko-WideSearch},一个由自动化合成与验证流水线构建的韩语广度搜索基准。每项任务指定一个集合父实体(如电视剧季、朝代、联赛、行政区、选举),要求完整列出成员并生成每项属性表格,评分采用项-列-行三重F1。该基准涵盖190个实体、228张表,分属十六类,设三个难度层级,通过两个独立调控的结构参数——表宽与二维复合键——使交叉成员匹配率从0%升至100%。金标准构建与评分共享同一规范化感知比较器,避免因格式问题误删稳定日期与计数列。对二十个网页智能体测试显示,其能恢复集合但无法完整获取行(如项F1 92.8 vs 行F1 53.7),准确率随参数强化持续下降,无论增加搜索次数或预算均无法弥补差距。细粒度分析表明,难点在于找到正确值而非格式化:开放文本字段错误最多,而日期、名称等有标准答案的字段通常正确。

原文摘要 · Abstract (English)

Web-agent benchmarks overwhelmingly measure depth -- pinning one obscure answer behind a chain of constraints -- while breadth, exhaustively enumerating a closed set and filling each item's attributes, is barely evaluated, especially outside English. Breadth is also hard to build: certifying that a gold set is complete and every cell correct is far costlier than checking a single answer. I introduce \textsc{Ko-WideSearch}, a Korean breadth-search benchmark built by an automated synthesize-and-verify pipeline. Each task names a set-parent entity -- a TV season, a dynasty, a league, an administrative region, an election -- and asks for its full membership plus a per-item attribute table, graded by Item-, Column-, and Row-F1. It spans 228 tables over 190 entities and sixteen categories across three difficulty tiers, set by two structural knobs I dial independently -- table width and a 2-D composite key -- so cross-product membership climbs from 0\% to 100\% across the tiers. A single normalization-aware comparator is shared between gold construction and grading, so stable date and count columns are not over-dropped on formatting alone. Across twenty web agents, the failure is consistent: agents recover the set but not the rows (e.g.\ Item-F1 92.8 against Row-F1 53.7), accuracy falls steadily as the knobs harden, and neither more search nor more spend closes the gap. Broken down by cell, the hard part is finding the right value, not formatting it: open-ended free-text cells fail most, while cells with a standard answer such as a date or a name usually come out right.

广度搜索韩语智能体评测数据枚举

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。