提出新指标衡量大模型支付系统流程执行的准确性。
Beyond Task Success: Measuring Workflow Fidelity in LLM-Based Agentic Payment Systems
- 用轨迹一致性评估模型执行顺序，分解为跳转召回与精确度。
- 18个模型中10个跳过支付确认环节，传统指标未发现此问题。
- 适合监管严格场景，帮助优化复杂工作流的可靠性。
基于大语言模型的多智能体系统在支付流程中应用日益广泛，但现有指标如任务成功率（TSR）和代理交接F1分数（HF1）仅关注最终结果或无序路由决策。本文提出代理成功度（ASR），一种基于轨迹一致性的度量方法，在转换层级比较实际与预期的代理执行序列，将性能分解为转换召回率与转换精确度。在涵盖18个大语言模型、9万次任务实例的支付多智能体系统（HMASP）上应用，ASR揭示其中10个模型在支付结账阶段系统性跳过确认检查点，而这一偏差在TSR和HF1下完全不可见；另有8个模型则完美执行该检查点。值得注意的是，GPT-4.1虽达成完美TSR与HF1，却存在隐蔽的工作流捷径；而GPT-5.2实现了完美ASR。基于ASR诊断进行提示优化与确定性路由防护，使部分原表现不佳模型的TSR提升最高达+93.8个百分点，证明轨迹级评估在受监管领域至关重要。
原文摘要 · Abstract (English)
LLM-based multi-agent systems are increasingly deployed for payment workflows, yet prevailing metrics, Task Success Rate (TSR) and Agent Handoff F1-Score (HF1), capture only final outcomes or unordered routing decisions. We introduce the Agentic Success Rate (ASR), a trajectory-fidelity metric that compares observed and expected agent execution sequences at the transition level, decomposing performance into Transition Recall and Transition Precision. Applied to the Hierarchical Multi-Agent System for Payments (HMASP) across 18 LLMs and 90,000 task instances, ASR reveals that 10 of 18 models systematically skip a confirmation checkpoint during payment checkout, a deviation invisible to both TSR and HF1, while 8 models enforce the checkpoint perfectly. Notably, GPT-4.1 exhibits hidden workflow shortcuts despite achieving perfect TSR and HF1, while GPT-5.2 achieves perfect ASR. Prompt refinements and deterministic routing guards guided by ASR diagnostics yield substantial TSR improvements, with gains up to +93.8 percentage points for previously struggling models, demonstrating that trajectory-level evaluation is essential in regulated domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。