评测大模型智能体是否能判断该用内知还是外工具，避免盲目调用。
From Knowing to Acting: Benchmarking Self-Awareness Capability of LLM Agents
- 分离判断（知道）与执行（行动），评估认知与行为是否一致。
- 开源模型在内部任务上过度依赖工具，成功率下降超40%。
- 适合研究模型自主性、工具使用策略的开发者和研究员。
外部工具的集成使大模型智能体从被动响应转向自主系统。然而，现有基准侧重执行成功，忽视了自知能力——即判断问题是否需外部资源或可用内部参数知识解决的能力。为此，我们提出KAPRO（Knowing-Acting Quadrant PRObe）框架，通过解耦元认知判断（知道）与自发执行（行动），评估认知-行为一致性。我们进一步构建了KAware数据集，严格将任务划分为外部、内部和混合三类，系统探测这些认识论边界。跨多种智能体架构的实验表明，自知能力与任务成功率强相关，但在仅依赖内部能力的任务中显著下降。开源及指令跟随型模型因浅层模式匹配，表现出更强的工具滥用倾向；而专有及推理导向型模型则展现出更可靠的认知门控机制。基准与代码见 https://github.com/AI-Santiago/KAware。
原文摘要 · Abstract (English)
The integration of external tools has transitioned LLM agents from passive responders to autonomous systems. However, current benchmarks prioritize execution success, neglecting self-awareness capability, the ability to discern whether a problem requires necessary external resources or can be solved via internal parametric knowledge. To address this, we introduce KAPRO (Knowing-Acting Quadrant PRObe), a framework that evaluates cognitive-behavioral alignment by decoupling an agent's metacognitive judgment (Knowing) from its spontaneous execution (Acting). We further construct KAware, a dataset rigorously partitioning tasks into external, internal, and hybrid subspaces to systematically probe these epistemic boundaries. Extensive experiments across diverse agent architectures show that self-awareness capability is strongly correlated with task success but degrades sharply in internal-capability settings. Moreover, open-source and instruction-following models exhibit stronger tool overuse due to shallow pattern matching, while proprietary and reasoning-oriented models demonstrate more reliable cognitive gating. Benchmark and codes are available at https://github.com/AI-Santiago/KAware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。