用自动机方法评估大模型智能体的完整执行路径，发现传统评价忽略的关键差异。
CORE: Full-Path Evaluation of LLM Agents Beyond Final State
- 基于确定性有限自动机建模任务合法调用路径
- 五项指标揭示代理在安全、效率上的真实表现差异
- 适合关注智能体行为细节与安全性的研究者使用
通过函数调用序列解决现实任务的AI智能体评估仍是开放挑战。现有评测常简化为对最终状态的二元判断，忽略了安全、效率和中间步骤正确性等关键方面。我们提出一种基于确定性有限自动机（DFAs）的框架，将任务编码为有效工具使用路径集合，实现对智能体行为在多种世界模型中的规范评估。在此基础上，我们引入CORE评测套件，包含五项指标：路径正确性、路径正确性-Kendall's tau综合度量、前缀关键性、有害调用率与效率，量化智能体对预期执行模式的对齐程度。在多种环境下的实验表明，该方法揭示了传统最终状态评估下看似等效的智能体之间的重要性能差异。
原文摘要 · Abstract (English)
Evaluating AI agents that solve real-world tasks through function-call sequences remains an open challenge. Existing agentic benchmarks often reduce evaluation to a binary judgment of the final state, overlooking critical aspects such as safety, efficiency, and intermediate correctness. We propose a framework based on deterministic finite automata (DFAs) that encodes tasks as sets of valid tool-use paths, enabling principled assessment of agent behavior in diverse world models. Building on this foundation, we introduce CORE, a suite of five metrics, namely Path Correctness, Path Correctness - Kendall's tau Composite, Prefix Criticality, Harmful-Call Rate, and Efficiency, that quantify alignment with expected execution patterns. Across diverse worlds, our method reveals important performance differences between agents that would otherwise appear equivalent under traditional final-state evaluation schemes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。