将工业遥测数据转化为可复现的智能体评估任务
BTS-AgentBench: A Deterministic, Replayable Pipeline from Read-Only Telemetry Logs to Agent Benchmarks
- 构建从只读遥测日志到多轮智能体任务的确定性转换流程
- 发布532条任务数据，训练/验证/测试集划分精确复现
- 支持可验证、可重复的智能体评估，适合工业AI研究者
工业现场包含大量只读遥测数据，但现有基准未明确如何将其转化为可执行的多轮智能体任务。本文提出BTS-AgentBench，一个将遥测元数据与原始历史记录标准化为只读工具库的流水线。该方法生成静态任务，基于工具推导出正确答案与证据，并将保留任务升级为类型化、边界限定的面向操作员的剧集。发布的532行数据包含任务澄清、目标修订、时间戳策略、质量门控报告和证据归属，同时保持源计算过程与划分一致。代码合同预检报告零问题，构造排除控制器处理0/532行无异常。两次独立的原始数据到剧集构建完全匹配11个逻辑工具库导出，并精确重现356/87/89的训练/验证/测试集成果。将该统一构建路径应用于XAI4HEAT，生成204个剧集；在其41行预留测试集上，控制器完成0行，而保留的GPT-5.5执行完成全部41行。代码、成果与复现报告见https://github.com/kjy7567/BTS-AgentBench。
原文摘要 · Abstract (English)
Industrial sites contain large volumes of read-only telemetry, but few benchmarks specify how to compile these records into executable multi-turn agent tasks. We present a telemetry-to-episode construction method instantiated as BTS-AgentBench. The pipeline normalizes BTS metadata and raw histories into a read-only tool store, compiles static tasks with tool-derived gold answers and evidence, and lifts retained tasks into typed, bounded operator-facing episodes. The 532-row release adds clarification, goal revision, timestamp policy, quality-gated reporting, and evidence attribution while preserving the source computation and split. Coded contract preflight reports zero findings, and the construction-exclusion controller completes 0/532 rows. Two independent raw-to-episode builds match all 11 logical tool-store exports and reproduce the released 356/87/89 train/dev/test artifact exactly. Applying the shared construction path to XAI4HEAT produces 204 episodes; on its 41-row held-out test split, the controller completes 0 rows and the retained GPT-5.5 execution completes all 41. Code, artifacts, and replay reports are available at https://github.com/kjy7567/BTS-AgentBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。