arXiv:2511.19957cs.CL2025-11被引 3

评测智能助手选对应用的能力,发现顶尖模型仍常选错。

AppSelectBench: Application-Level Tool Selection Benchmark

  • 构建真实用户任务生成流水线,覆盖百种常用软件
  • 百万级任务测试显示顶级模型仍频繁选错应用
  • 适合研究智能代理跨应用决策能力的学者和开发者

计算机使用代理(CUAs)越来越多地依赖外部工具来完成复杂现实任务。为有效运作,代理需具备应用选择能力——在调用细粒度工具(如API)前决定使用哪个应用程序。这关系到是否初始化正确环境、避免编排混乱并聚焦相关上下文。然而现有基准主要评估细粒度API选择,难以考察模型在不同应用间推理与选择的能力。为此,我们提出AppSelectBench,一个全面评估CUAs应用选择能力的基准。该基准包含新型用户任务生成流程,可规模化生成真实、多样且语义一致的用户意图;并提供统一评估协议,涵盖随机、启发式、零样本、少样本及检索增强等设置。AppSelectBench涵盖100种广泛使用的桌面应用,包含超过10万条真实、多样且语义基础的用户任务。对闭源与开源大模型的广泛实验揭示了其在跨应用推理中的系统性优劣势,表明即使最先进模型仍难以做出一致的应用选择。这些结果确立AppSelectBench作为研究和推动智能代理应用级推理这一关键但未被充分探索能力的基础。代码已开源:https://microsoft.github.io/appselectbench/

原文摘要 · Abstract (English)

Computer Using Agents (CUAs) are increasingly equipped with external tools, enabling them to perform complex and realistic tasks. For CUAs to operate effectively, application selection, which refers to deciding which application to use before invoking fine-grained tools such as APIs, is a fundamental capability. It determines whether the agent initializes the correct environment, avoids orchestration confusion, and efficiently focuses on relevant context. However, existing benchmarks primarily assess fine-grained API selection, offering limited insight into whether models can reason across and choose between different applications. To fill this gap, we introduce AppSelectBench, a comprehensive benchmark for evaluating application selection in CUAs. AppSelectBench contains a novel user task generation pipeline that produces realistic, diverse, and semantically grounded user intents at scale, together with unified evaluation protocols covering random, heuristic, zero-shot, few-shot, and retrieval-augmented-settings. AppSelectBench covers one hundred widely used desktop applications and includes more than one hundred thousand realistic, diverse, and semantically grounded user tasks. Extensive experiments across both closed-source and open-source large language models reveal systematic strengths and weaknesses in inter-application reasoning, showing that even the most capable models still struggle to make consistent application choices. Together, these results establish AppSelectBench as a foundation for studying and advancing application level reasoning, an essential yet underexplored capability of intelligent CUAs. The source is available at https://microsoft.github.io/appselectbench/.

智能代理应用选择评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。