测试智能体日志能否定位故障根源，发现日志不足时模型易误判。
TelemetrySuffBench: Is Agent Telemetry Sufficient for Failure-Origin Diagnosis?
- 构建可控基准测试，分离故障检测、定位与安全回避三任务。
- 完整日志下定位准确率最高达97.2%，但多数日志仅能检测故障无法定位。
- 模型依赖日志中的决策内容和溯源信息，缺乏时会错误回答或不回避。
智能体系统日益暴露执行轨迹，但揭示故障的日志仍不足以定位故障源头。我们提出TelemetrySuffBench，一个受控基准，用于分离故障检测、故障源定位与证据不足时的安全回避。该基准构建包含延迟绑定故障的标准化多组件轨迹，生成成对的粗粒度视图、七因素日志掩码及完全等价的模糊起源对。我们采用统一协议评估五种前沿语言模型，包含明确候选集、无效输出统计、子组分析与冻结盲测。在完整日志下，起源步骤的Top-1准确率在33.8%至97.2%之间。元数据、OpenTelemetry兼容及OpenInference兼容视图保留99.5%至100%的检测F1，但起源定位准确率不超过0.5%，暴露出强检测弱定位的差距。因子消融表明：移除决策内容使所有模型定位准确率归零；移除溯源信息则导致模型依赖的显著损失。在需回避的复杂模糊输入上，证据门控使三个模型的非支持唯一起源回答降低12.5至48.6个百分点，而两个模型仍持续作答，显示安全回避能力存在显著模型差异。冻结盲测结果重现了同一生成器家族内的核心模式。结果表明，终端状态可支持检测，但可靠因果归因需显式决策到溯源链接及跨模型有效的回避机制。数据集与基准实现已公开于https://anonymous.4open.science/r/TelemetrySuffBench-E635/README.md。
原文摘要 · Abstract (English)
Agent systems increasingly expose execution traces, yet telemetry that reveals a failure may still be inadequate for identifying where that failure originated. We introduce TelemetrySuffBench, a controlled benchmark that separates failure detection, fault-origin localization, and safe abstention under insufficient evidence. The benchmark constructs canonical multi-component traces with delayed-binding faults and renders them as paired coarse views, seven-factor telemetry masks, and exact-equal ambiguous origin pairs. We evaluate five frontier language models using unified protocols, explicit candidate sets, invalid-output accounting, subgroup analyses, and a frozen blind holdout. With full telemetry, origin-step Top-1 accuracy ranges from 33.8% to 97.2% across models. Metadata, OpenTelemetry-compatible, and OpenInference-compatible views retain 99.5% to 100% detection F1 while limiting origin-step accuracy to at most 0.5%, exposing a robust detection-localization gap. Factor ablations further show that removing decision content reduces origin-step accuracy to zero for every model, while provenance removal also causes large model-dependent losses. On rich ambiguous inputs that require abstention, evidence gating reduces unsupported unique-origin answers by 12.5 to 48.6 percentage points for three models, whereas two models still answer every case, revealing strong model dependence in safe abstention. Results on the frozen holdout reproduce the central pattern within the same generator family. These findings show that terminal status can support detection, whereas reliable causal attribution requires explicit decision-to-provenance links and abstention safeguards that remain effective across models. The dataset and benchmark implementation are available at https://anonymous.4open.science/r/TelemetrySuffBench-E635/README.md.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。