构建首个专家级商品编码推理基准,评估模型在复杂规则下的深层搜索能力。
HSCodeComp: A Realistic and Expert-level Benchmark for Deep Search Agents in Hierarchical Rule Application
- 基于真实电商数据设计层级规则推理任务,预测10位海关编码
- 顶尖模型准确率仅46.8%,远低于人类专家的95.0%
- 揭示规则模糊性与层次结构对模型的挑战,适合评估高阶推理系统
有效的深度搜索代理不仅需获取开放域和领域特定知识,还需应用复杂规则(如法律条文、医学手册和关税规则)。这些规则常具有模糊边界和隐含逻辑关系,使精准应用极具挑战,但当前代理评估基准普遍忽视此能力。为此,我们提出HSCodeComp,首个面向真实场景、具备专家水平的电子商务基准,用于评估代理在层级规则应用中的表现。该任务要求代理依据规则推断产品10位《协调制度编码》(HSCode),编码由世界海关组织制定,对全球供应链效率至关重要。数据集源自大规模电商平台的真实数据,包含632个产品条目,覆盖多样化品类,均由多名人工专家标注。对多个先进大模型及开源/闭源代理的实测显示:最佳模型仅达46.8%的10位准确率,远低于人类专家的95.0%。详细分析表明层级规则应用存在显著挑战,且测试时扩展无法进一步提升性能。
原文摘要 · Abstract (English)
Effective deep search agents must not only access open-domain and domain-specific knowledge but also apply complex rules-such as legal clauses, medical manuals and tariff rules. These rules often feature vague boundaries and implicit logic relationships, making precise application challenging for agents. However, this critical capability is largely overlooked by current agent benchmarks. To fill this gap, we introduce HSCodeComp, the first realistic, expert-level e-commerce benchmark designed to evaluate deep search agents in hierarchical rule application. In this task, the deep reasoning process of agents is guided by these rules to predict 10-digit Harmonized System Code (HSCode) of products with noisy but realistic descriptions. These codes, established by the World Customs Organization, are vital for global supply chain efficiency. Built from real-world data collected from large-scale e-commerce platforms, our proposed HSCodeComp comprises 632 product entries spanning diverse product categories, with these HSCodes annotated by several human experts. Extensive experimental results on several state-of-the-art LLMs, open-source, and closed-source agents reveal a huge performance gap: best agent achieves only 46.8% 10-digit accuracy, far below human experts at 95.0%. Besides, detailed analysis demonstrates the challenges of hierarchical rule application, and test-time scaling fails to improve performance further.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。