arXiv:2605.28876cs.SEcs.AI2026-05

对比11种日志压缩工具,发现混合方法在成本与效果间最优。

LogDx-CI: Benchmarking Log Reduction Tools for LLM Root-Cause Diagnosis

论文配图:LogDx-CI: Benchmarking Log Reduction Tools for LLM Root-Cause Diagnosis
图 1 · 摘自论文原文
  • 用混合grep+tail方法在极低开销下保留关键诊断信息
  • 同一工具在智能代理循环中表现差距缩小7倍,但成本差异仍存
  • 跨模型组合(gpt-5-mini+Claude Haiku)性能超同类组合0.071分

CI失败日志规模大(中位数5000行,最大20万行)且噪声多。依赖上游工具将日志缩减为可管理上下文的编码代理,但领域内缺乏对不同缩减方法是否保留足够证据的公开实证比较。我们提出LogDx-CI基准,对比11种上下文缩减工具(原始、尾部、grep、三种RTK模式、两种真实LLM map-reduce摘要器、三种混合路由器)在35个真实GitHub Actions失败案例上的表现,由3类LLM调试器(Claude Haiku 4.5、Claude Sonnet 4.6、OpenAI gpt-5-mini)及一个Sonnet 4.6工具使用代理评分。报告三项核心发现:(1) 混合grep+tail路由器在成本-质量帕累托前沿占优;前两名方法每案例约$0.03成本,得分0.670/0.666,与独立grep相当,但仅需其4.5倍的词元量;(2) 在代理循环场景中,不同工具间质量差距缩小7倍(单次推理跨度0.42→代理循环跨度0.059);代理通过后续工具调用修复弱上下文,但弱上下文仍导致代理调用次数增加2–4倍;(3) 跨家族摘要-调试组合(gpt-5-mini摘要器+Claude Haiku调试器)在四种变体上平均优于同家族组合0.071分,推翻了自调用偏见假设;gpt-5-mini摘要器在代理循环中排名第一(得分0.749),每案例仅0.37次工具调用,且缩减成本仅为Haiku摘要器的1/10($0.18 vs $1.75/案例)。所有数据、代码、案例包及可复现性基础设施均公开。

原文摘要 · Abstract (English)

CI failure logs are large (median 5k lines, max 200k in this corpus) and noisy. Coding agents that try to debug them depend on an upstream tool to reduce the log to a manageable context, but the field has had no public empirical comparison of which reductions preserve enough evidence for downstream LLM diagnosis. We introduce LogDx-CI, a benchmark that compares 11 context-reduction tools (raw, tail, grep, three RTK modes, two real LLM map-reduce summarizers, three hybrid routers) on 35 real GitHub Actions failure cases, scored by 3 LLM debugger families (Claude Haiku 4.5, Claude Sonnet 4.6, OpenAI gpt-5-mini) plus a Sonnet 4.6 tool-using agent. We report three load-bearing findings. (1)~Hybrid grep+tail routers dominate the cost-quality Pareto frontier; the top two methods score 0.670 / 0.666 at $\sim$ \$0.03 per case, same-ballpark quality as standalone grep at $4.5\times$ fewer tokens. (2)~In the agent-loop regime, the quality range across reduction tools collapses $7\times$ (single-shot spread 0.42 $\to$ agent-loop spread 0.059); the agent rescues weak contexts via follow-up tool calls. However, cost differences persist: weak contexts force the agent to issue 2--4$\times$ more tool calls to recover. (3)~A cross-family LLM-summary pair (gpt-5-mini summarizer feeding a Claude Haiku debugger) beats the same-family pair by $+0.071$ averaged across four diagnoser variants, falsifying the self-call-bias hypothesis on this task. The gpt-5-mini summarizer is also the agent-loop \#1 method (score 0.749) at $0.37$ tool-calls per case and $10\times$ lower reducer cost than the Haiku summarizer (\$0.18 vs \$1.75 per case). All data, code, per-case bundles, and reproducibility infrastructure are public.

日志分析LLM调试工具评估效率优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。