用方差分析三维度监控，提前发现代理系统结构缺陷。
Monitoring Agentic Systems Before They're Reliable
- 按运行内、跨运行、结构三个层面，用方差识别故障类型。
- 结构监控零误差（CV=0.00），任务错误被掩盖无法检测。
- 97%问题可自动处理，仅2%需人工介入，适合监管行业使用。
进入生产的代理系统通常为部分集成的组合体，其故障主要源于结构缺陷而非任务错误。此时任务级错误检测可能失效：结构故障会遮蔽任务监控本应捕捉的信号。本文提出一种监测与分类方法，将代理系统评估分解为质量、适用性、效率三个维度，并在三个监测层级（运行内、跨运行、结构）上应用方差作为表征信号。结果依据改编自FMEA的严重性分类进行路由，集中人力关注需调查的部分。在包含120个文档包、220次运行的合成测试平台上验证：运行内监控揭示确定性阶段缺陷（CV=0.02），跨运行监控揭示随机集成后果（CV=1.25，L2层占比24%），结构监控则以完美一致性（CV=0.00）识别出集成缺口。注入的任务级错误与干净基线无异，证实结构缺陷会掩盖任务信号。确定性分类使97%发现可自动化追踪，仅2%变量行为留待人工处理。我们提出在第一阶段证据基础上，建立成熟度分级模型，随集成缺陷消除，监测从结构表征过渡到错误检测再到可靠性追踪。该分类体系、基于方差的层级表征及严重性模型可迁移至受监管行业的文档驱动多阶段代理流程；具体校准需依领域定制。早期部署监测：它首次发现的问题，正是最需修复的。
原文摘要 · Abstract (English)
Agentic systems entering production typically operate as partially integrated assemblies where structural defects, not task-level errors, dominate the failure landscape. At this maturity level, task-level error detection may be infeasible: structural failure modes mask the signal that task-level monitors are designed to detect.We present a monitoring and triage methodology that decomposes agentic system evaluation into three dimensions (quality, suitability, efficiency) at three monitoring scopes (within-run, cross-run, structural), using variance as a characterization signal. Findings are routed through severity classification adapted from FMEA, concentrating human attention on the subset that warrants investigation. We evaluate on a synthetic testbed of 220 runs across 120 document bundles with controlled error injection.Three results emerge. Monitor scope determines failure type: within-run monitors surface deterministic stage defects (CV = 0.02), cross-run monitors surface stochastic integration consequences (CV = 1.25, 24% at L2), and a structural monitor identifies an integration gap with perfect consistency (CV = 0.00). Injected task-level errors are indistinguishable from clean baselines, confirming structural defects mask task-level signal. Deterministic triage routes 97% of findings to automated tracking, leaving the 2% reflecting variable behavior for human investigation.We propose, on Stage 1 evidence, a maturity-staging model in which monitoring transitions from structural characterization to error detection to reliability tracking as integration defects resolve. The taxonomy, CV-based scope characterization, and severity model transfer architecturally to document-driven, multi-stage agentic workflows in regulated industries; specific calibrations are domain-specific. Deploy monitoring early: the first thing it finds is the most important thing to fix.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。