用有限状态机解析大模型智能体行为，实现失败与下一步预测。
Automata from Agent Traces: Failure and Next-Step Prediction
- 将多步任务轨迹压缩为紧凑的有限状态机，捕捉跨运行行为拓扑。
- 在12个数据集上复现精度超0.997，失败预测AUROC达0.94。
- 适合安全审计与实时监控，支持早期终止故障运行。
基于大语言模型的智能体执行多步骤任务，但其行为结构不透明：长而无序的轨迹难以满足部署所需的安全部署审核与运行时监控。现有方法仅针对单条轨迹或成功案例，忽略了连接下一步预测与失败预测的跨运行拓扑结构。为恢复这一共享结构，我们把整个轨迹语料库压缩为一个单一、紧凑的有限状态机（FSM），作为大模型智能体不可预测行为的结构基础。在十二个公开数据集上，该 FSM 状态数为7-43，对保留数据的复现精度≥0.997，且不同划分间拓扑高度一致，构建时间以毫秒计。该结构同时支持两项预测目标：在所有匹配真实路径的数据集上，基于 FSM 状态的上下文比 Agent Workflow Memory 更优；对于失败预测，每个状态的行为特征在保留数据上达到最高0.94的 AUROC，且在线监控器可从部分轨迹中将失败运行排在成功之前，提前触发终止。行为拓扑更受部署环境影响而非大模型本身，提供了一种模型无关的结构化原语，用于安全审计与运行时监控。
原文摘要 · Abstract (English)
LLM-based agents execute multi-step tasks, but their behavioral structure remains opaque: long unstructured traces resist the safety auditing and runtime monitoring that deployment requires. Existing approaches operate per-trace or success-only, so they miss the cross-run topology that links next-step and failure prediction. To recover that shared structure, we collapse an entire trace corpus into a single, compact finite-state machine (FSM) that serves as a structural substrate for the otherwise unpredictable behavior of LLM agents. Across twelve public datasets, the FSMs are compact (7-43 states), replay held-out data at >=0.997 fitness with near-identical topology across splits, and build in milliseconds. This substrate addresses both prediction goals. For next-step prediction, FSM-state context outperforms Agent Workflow Memory on every ground-truth-matched dataset. For failure prediction, per-state behavioral features reach held-out AUROC up to 0.94, and an online monitor ranks failing runs above passing ones from a partial trace, triggering early stopping well before completion. Behavioral topology thus appears shaped more by the deployment harness than by the LLM, providing a model-agnostic structural primitive for safety auditing and runtime monitoring.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。