评测大模型在分布式系统中的代码修复能力，发现上下文信息显著提升效果。
Evaluating Agentic Code Repair Capabilities in Distributed Systems
- 构建60个真实分布式系统漏洞的修复基准DDBench，分三难度等级
- 有调试上下文时整体修复成功率提升18.1个百分点
- 弱模型靠上下文提效果，强模型则更高效，需谨慎设计上下文
基于大模型的编码智能体在单进程SWE任务上进步迅速，前沿模型在SWE-bench Verified上已达到高70分段。然而，分布式系统调试仍属研究空白：错误跨越进程、节点与协议交互，根源常无法仅从源码还原，且非确定性调度下的暴力探索不可行。现有评估存在两大缺口：缺乏针对分布式系统漏洞的代码修复基准，也无受控实验分离外部调试上下文对智能体表现的影响。本文提出DDBench，包含从13个开源分布式系统中挖掘出的60个历史漏洞，分为三个难度层级。每个案例在两种匹配条件下评估：仅提供错误症状和代码库的“症状仅限”条件，以及额外提供受限调试上下文（日志、追踪、运行时状态及针对性代码调查笔记）的“上下文增强”条件，以隔离上下文影响。对十种LLM在DDBench上的评估显示：第一，分布式调试揭示了单进程基准未暴露的推理维度，模型通过率跨度达61个百分点，15组顶尖模型对中有9组在最难题集上差异显著（p < 0.05）；第二，有限调试上下文使总体通过率提升+18.1个百分点，且提升不对称——弱模型获通过率提升，强模型获效率提升；第三，调试上下文需精心构建，即使忠实记录也可能误导大模型。
原文摘要 · Abstract (English)
LLM-based coding agents have advanced rapidly on single-process SWE tasks, with frontier models now clustering in the high-70s on SWE-bench Verified. Distributed-system debugging, however, remains an under-explored regime: bugs span processes, nodes, and protocol interactions, with root causes rarely recoverable from source alone and brute-force exploration intractable across non-deterministic interleavings. This leaves two gaps in LLM and agent evaluation: no code-repair benchmark targets distributed-system bugs, and no controlled study isolates how much externally provided debugging context changes agent success on them. We introduce DDBench, a code-repair benchmark of 60 historical bugs mined from 13 open-source distributed systems, partitioned into three difficulty tiers. DDBench evaluates every case under two matched conditions: a symptom-only condition where the agent receives only the bug symptom and repository, and a context-augmented condition where it additionally receives a bounded debugging context (logs, traces, runtime state, and targeted code-investigation notes), isolating the effect of debugging context from model capability. The evaluation of ten LLMs on DDBench reveals several findings. First, distributed debugging exercises a reasoning dimension that single-process benchmarks do not surface: models' pass rates span 61 pp, and pairwise bootstrap separates 9 of 15 top-tier model pairs at p < 0.05 on DDBench's hardest case-set. Second, bounded debugging context lifts aggregate pass rate by +18.1 pp, and the lift is asymmetric: weaker models gain pass rate, while stronger models gain efficiency. Third, debugging context requires careful curation, as even faithful debugging context can sometimes mislead LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。