AI系统评估盲区导致故障无声传播,从训练到部署均可能失效却无预警。
Evaluation Blindness: How Silent Measurement Failures Corrupt AI Systems from Training to Deployment

- 提出评估盲区概念:故障时指标仍显示正常,无法察觉异常。
- 实证发现53%公开故障案例为无声失败,六类生产故障中100%不可监测。
- 建立统一检测框架与失败预算机制,强调测量系统需贯穿全生命周期。
AI系统可能无声失效,故障在训练循环、评估管道和生产监控中层层传递,直至造成下游危害才被察觉。本文提出‘评估盲区’:当测量函数M在系统实际失效时仍输出健康状态读数,且无辅助信号提示差异,则存在评估盲区。该问题出现在训练与部署两个阶段,文献常分开处理。训练期,奖励模型被操纵,重要性采样修正无声错误,基准污染虚增微调评估,而损失曲线看似正常,梯度更新照常进行。部署期,监控无法捕获六类生产故障,其中一类因结构定义完全无声。本文提出统一可检测性谓词。四个训练期案例研究揭示具体崩溃,包括TRL PR #6594中的真实实现漏洞——损失下降时梯度已被污染。基于50个来自法院文件与监管报告的真实事件,验证六类分类法,发现53%可验证的公开故障为沉默型。失败预算框架将可接受失败率与使用场景风险等级挂钩。结论明确:测量基础设施是贯穿全生命周期的正确性关键,不仅限于评估阶段。数据、代码与分类模板见https://github.com/priyanka25aug/llm-failure-taxonomy。
原文摘要 · Abstract (English)
AI systems can fail silently. The failure propagates through training loops, evaluation pipelines, and production monitoring stacks until downstream harm makes it visible. This paper introduces evaluation blindness: a measurement function M exhibits evaluation blindness with respect to failure class F when it produces readings indistinguishable from a healthy state while the system is actually failing, with no auxiliary signal flagging the gap. The problem surfaces at two lifecycle stages the literature has treated separately. At training time, reward models are gamed, importance-sampling corrections are silently miscalculated, and benchmark contamination inflates fine-tuning evaluations, all while loss curves look healthy and gradient updates proceed normally. At deployment time, monitoring fails to catch six classes of production failure, including an Operational category that is 100% silent by structural definition. We provide a formal detectability predicate unifying both stages. Four training-time case studies trace concrete breakdowns, including a real implementation bug in TRL PR #6594 where gradients are corrupted as loss decreases normally. A six-class taxonomy validated against 50 real-world incidents from court documents and regulatory filings finds that 53% of verifiable public failures were silent. A failure budget framework ties acceptable failure rates to use-case risk class. The implication is direct: measurement infrastructure is a correctness concern across the full AI lifecycle, not just at evaluation time. Data, code, and taxonomy schema are at https://github.com/priyanka25aug/llm-failure-taxonomy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。