测试大模型在真实线上故障排查中的表现，发现其准确率不足三成。
ORCA-bench: How Ready Are Language Model Agents for Oncall?
- 构建真实生产环境的故障排查基准，含六天完整日志与代码访问。
- 顶尖模型在中等难度任务中准确率仅25.3%，硬任务下低至10.0%。
- 模型依赖源码，且错误率高，反映当前智能运维仍不成熟。
大型语言模型能写代码、修漏洞、查资料，但线上故障排查（RCA）要求从模糊用户报告出发，推理噪声干扰的指标、日志、链路追踪和源码，常在事件发生数小时后进行。我们提出 ORCA-bench，一个贴近真实生产的通用编码代理评估基准。该基准结合一个实时接入 OpenTelemetry 的微服务系统，通过 Prometheus、Jaeger、OpenSearch（Grafana 可视化）提供六天的指标、日志与追踪数据，并开放全量源码。共包含 1,079 个故障排查任务，系统性地变化报告清晰度、发现延迟及并发故障场景。真值症状由资深 SRE 标注并确认，模型评分由人类独立复评（Cohen's $κ_w=0.90$）。五款前沿代理中，最佳在中等难度任务上准确率为 25.3%，硬任务为 10.0%，即便使用 Claude Fable 5 仍存在明显差距。最弱模型在 40% 报告中生成不合理的根因，移除源码访问导致所有指标下降。这些结果基于 50 GB / 六天的精调测试集，任务独立评估，系统代码与监控公开。由于真实生产系统规模更大、动态更强、特性更独特，当前表现仅为工程投入的下限。我们已开源数据集：https://hub.harborframework.com/datasets/orca-bench/orca-bench。
原文摘要 · Abstract (English)
Large language models can write, patch, and search code, but oncall root cause analysis (RCA) demands something different: reasoning over noisy metrics, logs, traces, and source code, starting from ambiguous user-facing reports, often hours after the incident began. We introduce ORCA-bench, a benchmark that puts general-purpose coding agents in a production-fidelity oncall setting. ORCA-bench pairs a live OpenTelemetry-instrumented microservice system--exposing six days of metrics, logs, and traces through real telemetry interfaces (Prometheus, Jaeger, and OpenSearch via Grafana) and full source-code access--with 1,079 RCA tasks that systematically vary report specificity, time-to-detection, and co-occurring fault scenarios. Ground-truth symptoms are curated and signed off by expert SREs, and our LLM-as-judge is independently re-scored by humans (Cohen's $κ_w=0.90$). Across five frontier agents, the best RCA Accuracy is 25.3% on Medium-difficulty tasks (the realistic-input setting) and 10.0% on Hard--a gap that remains even with Claude Fable 5. The weakest model hallucinates an implausible root cause in 40% of incident reports, and removing source-code access degrades every metric. Crucially, these are performances on a curated 50 GB / six-day testbed with tasks investigated in isolation on a system whose code and instrumentation are public. Since real production systems are order of magnitudes larger, more dynamic, and more idiosyncratic, the gap we report is a lower bound on the engineering investment required before frontier coding agents can be safely entrusted with production reliability. We release the public set at https://hub.harborframework.com/datasets/orca-bench/orca-bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。