测试智能体能否识别无法完成的任务，发现多数模型会盲目执行。
Do Agents Know What They Can't Do? Evaluating Feasibility Awareness in Tool-Using Agents
- 用关键工具缺失法自动构造不可行任务
- 9个模型中错误继续率最高达73.9%
- 多智能体架构能有效减少错误执行
使用工具的智能体常因冗长推理链和反复调用工具而产生高昂计算开销。在实际场景中，许多任务在受限工具环境下不可行，即完成任务所需能力缺失。及时识别不可行任务并提前终止可显著降低无效开销。本文提出FeasiGen，一个自动构建不可行任务的流水线：通过分析多个智能体系统中成功执行的工具调用轨迹，识别在不同策略下一致出现的关键工具，并将其屏蔽，从而将可解任务转为不可行任务。人工验证显示，生成任务的不可行性标注准确率超过94%。我们进一步引入可行性感知评估指标，衡量智能体识别不可行任务并合理终止的能力。九个模型的广泛评估表明，其可行性检测能力普遍较弱，错误继续率高达73.9%。此外，多智能体架构在不可行条件下显著降低了错误执行。
原文摘要 · Abstract (English)
Tool-using agents often incur substantial computational cost due to long reasoning chains and iterative tool usage. In practical scenarios, many tasks become infeasible under constrained tool environments, where the capabilities required for successful task completion are unavailable. Detecting infeasible tasks and stopping execution early can significantly reduce unnecessary execution cost. In this work, we propose FeasiGen, an automatic pipeline for constructing infeasible agent tasks by identifying the critical tools required for successful task completion. Our approach extracts tool-calling traces from successful executions across multiple agent systems, identifies critical tools consistently shared across diverse execution strategies, and masks these tools to automatically transform solvable tasks into infeasible ones. Human verification confirms that the infeasibility annotations for our constructed tasks achieve over 94% accuracy. We further introduce feasibility-aware evaluation metrics for measuring whether agents can recognize infeasible tasks and stop execution appropriately. Extensive evaluations across nine models reveal substantially weak infeasibility detection ability, with false continue rate reaching up to 73.9%. We further observe that multi-agent architectures significantly reduce erroneous execution under infeasible conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。