用知识图谱预测调试关键事件,零成本定位故障点。
Knowledge-Based Zero-Replay Debugging of Multi-Agent LLM Traces
- 构建事件知识图谱,融合路由、记忆、工具调用等结构信息
- 在37个轨迹族上将定位准确率从0.73提升至0.93,无需实际重放
- 适合大规模多智能体LLM系统可靠性调试,成本低且可审计
多智能体大语言模型系统的可靠运行依赖于对长执行轨迹的调试,但关键因果事件常深埋于无结构的日志中。传统反事实重放(回溯、修改、重跑)成本随候选事件数线性增长,难以规模化应用。本文将调试问题建模为基于知识的决策支持任务:将每个轨迹编译为涵盖路由、记忆、工具使用、不确定性与潜在证据的结构化事件知识图谱,并训练校准的预测器,决定稀缺重放预算应投向何处。我们不提供新重放机制,而是提出一种无需重放即可预测重放结果的方法——零重放反事实效应预测:在固定预算下,预测重放引擎会标记为高影响的事件。BranchPoint-Latent 是一个轻量级预测器,基于可观测、结构、不确定性和潜在特征构建。经37个轨迹家族的确定性重放验证,单一学习排序梯度提升预测器在未见家族上将每轨迹定位准确率(Branch Recall@5)从0.73提升至0.93,实现零重放开销。本文不宣称通用最优,而是明确界定何时简单图中心性足够,何时需依赖学习到的证据。最终系统具备可审计性与成本效率,在可复现的成果基础上定位在成本-精度权衡前沿。
原文摘要 · Abstract (English)
Reliable operation of multi-agent large language model (LLM) systems depends on debugging long execution traces, where the few causally decisive events are buried in unstructured logs of messages, routes, memory writes, and tool calls. The standard tool is counterfactual replay (rewind, edit, and re-run the trajectory to measure each event's effect), but its cost grows linearly with the number of candidate events, making exhaustive replay infeasible at scale. We frame trace debugging as a knowledge-based decision-support problem. Each trace is compiled into a structured event knowledge graph over routing, memory, tool-use, uncertainty, and latent evidence, and a calibrated predictor decides where a scarce replay budget should be spent. We do not propose a new replay oracle; we propose a method to predict its results without paying the replay cost. We formulate zero-replay counterfactual-effect prediction: given a trace under a fixed budget, predict which events the oracle would mark high-effect before any replay is performed. BranchPoint-Latent is a lightweight predictor over observable, structural, uncertainty, and latent features of the knowledge graph. Calibrated against a deterministic replay oracle across 37 trace families, a single learning-to-rank gradient-boosted predictor raises per-trace localization (Branch Recall@5) from 0.73 to 0.93 on held-out families at zero oracle-replay cost. Rather than claiming universal dominance, we characterize when cheap graph centrality suffices and when learned evidence is necessary. The result is an auditable, cost-efficient decision-support system for AI-reliability debugging, positioned explicitly on the cost-accuracy frontier with reproducible artifacts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。