构建首个带因果链标注的根因分析基准，揭示大模型诊断常‘有结论无依据’。
OpenRCA 2.0: From Outcome Labels to Causal Process Supervision
- 通过故障注入反推因果链，实现从结果标签到过程监督的升级
- 11个前沿大模型平均仅20.7%能准确找出全部根因
- 多数模型能猜对根因但无法验证其传播路径，适合评估可信推理能力
根因分析（RCA）是对大语言模型智能体能力的全面考验，涉及长上下文理解、多步推理和工具使用。然而现有数据集存在根本缺陷：仅标注根因，未提供从根因到症状的因果传播路径，使任务简化为简单的模式匹配。为支持严谨评估，我们提出PAVE步骤化标注协议，利用已知故障注入干预重建因果传播路径。该机制采用正向验证：从原因推导结果，而非逆向推理症状。应用PAVE构建了OpenRCA 2.0（500个实例），首个跨系统且具备步骤级因果标注的RCA基准。在11个前沿大模型上，准确恢复完整根因集合的成功率仅为20.7%。进一步放宽标准后发现‘无根据诊断’现象：模型在76.0%案例中识别出至少一个正确根因服务，但仅在61.5%中将其与症状的验证因果路径关联。仅依赖结果标签的评估会掩盖这一失败模式；步骤级因果真值是构建可信大模型根因分析代理的关键缺失环节。
原文摘要 · Abstract (English)
Root cause analysis (RCA) poses a holistic test of LLM agentic capabilities, such as long-context understanding, multi-step reasoning, and tool use. However, existing datasets suffer from a fundamental gap: they label only the root cause, not the propagation path connecting it to the observed symptom, which largely simplifies the task to naive pattern matching. To support rigorous evaluation, we introduce PAVE, a step-wise labeling protocol that leverages known interventions from fault injection to reconstruct causal propagation paths. The mechanism is forward verification: reasoning from cause to effect rather than inferring backward from symptoms. Applying PAVE yields OpenRCA 2.0 (500 instances), the first cross-system RCA benchmark with step-wise causal annotations for LLM agents. Across 11 frontier LLMs, recovering the exact root-cause set succeeds in only 20.7% of cases on average. To locate where this difficulty lies, we relax the criterion and find what we call the ungrounded diagnosis: agents identify at least one correct root-cause service in 76.0% of cases, but ground that service in a verified causal propagation path to the observed symptom in only 61.5%. Outcome-only evaluation hides this failure mode; step-wise causal ground truth is the missing piece for trustworthy LLM-based RCA agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。