为自治Kubernetes代理设计可验证的测试框架,解决结果不可复现的问题。
A measurement substrate for agentic Kubernetes operations: Methodology and a case study in retrieval-compounding falsification
- 构建闭环测试系统,主动注入故障并量化代理响应表现
- 发现三个干扰因素:索引缺陷、选择偏差超19%、小样本夸大效果三倍
- 开源工具支持可重复实验,适合可信自动化系统研究者使用
关于自主Kubernetes操作代理的实证主张大多无法被验证。现有工作缺乏与关闭代理基线的对照实验,存在严重选择偏差,且无预先注册的决策矩阵,样本量通常不足以克服评分系统的噪声。根本原因在于:代码代理有验证底座可快速生成“是否成功”的真值信号,而运维领域无此机制。本文提出agent-breakage——一种闭环测量框架,向目标Kubernetes集群注入故障,观察代理响应,在四个维度上基于真值评估响应,并积累带标签的(状态,动作,结果)三元组。该框架能区分框架错误与推理错误,通过确定性嵌入机制实现真正离条件控制,并强制执行预注册决策矩阵。以检索过往事故报告是否提升代理能力为例进行案例研究,发现三个混杂因素:pgvector索引缺陷、+19%的选择偏差、小样本估计使效果夸大近3倍。检索结果本身部分证伪:在3个密集语料场景中仅1个显著(p<0.05),合并效应仅+3.9个百分点,样本n=60时不显著。在360次跨场景密度扫描中显示,近邻的机制对齐优于单纯数量。框架已开源。
原文摘要 · Abstract (English)
Empirical claims about autonomous Kubernetes operations agents are largely unfalsifiable. Published work reports observational results without controlled comparisons against an agent-disabled baseline, selection bias is endemic, pre-registered decision matrices are absent, and samples are typically too small for the noise level of the underlying scoring system. The cause is the same gap that limits the agents themselves: code agents have a verification substrate that turns "did it work" into a fast, falsifiable, ground-truth signal, and operations has nothing equivalent. We present agent-breakage, a closed-loop measurement framework that injects faults into a target Kubernetes cluster, observes how an autonomous agent responds, scores the response on four axes against ground truth, and accumulates outcome-labeled (state, action, outcome) tuples. The framework distinguishes framework error from reasoning error, supports a true off-condition control via a deterministic-embedder mechanism, and enforces pre-registered decision matrices. We use it as a case study to test whether retrieval over past postmortems compounds an agent's capability. The methodological payload is three confounds the substrate caught during that case study, each of which would have produced a wrong published claim on a less instrumented version of the same work: a pgvector index bug, a +19% selection-bias artifact, and small-sample estimates that overstated effects by roughly 3x. The retrieval result itself is a partial falsification: 1 of 3 dense-corpus scenarios significant at p<0.05, pooled effect +3.9 percentage points, not significant at n=60. A within-scenario corpus-density sweep at 360 runs shows that mechanistic alignment of near-neighbors dominates raw count. The framework is released open source.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。