动态生成网络自动化测试用例，让AI代理真实表现更可信
NetArena: Dynamic Benchmarks for AI Agents in Network Automation
- 用动态生成框架统一抽象各类网络任务，按需生成无限测试查询
- 实测显示多数代理在复杂场景下性能仅13%-38%，最低达3%
- 不仅能测正确性，还能捕捉安全性和延迟等关键行为
随着AI代理进入网络系统运维等高风险领域，评估其实际可靠性愈发重要。现有基准因设计静态、数据集规模小而存在统计方差大、易被污染的问题，且难以反映生产环境的复杂性。我们提出NetArena，一个面向网络应用的动态基准生成框架。NetArena引入新颖的抽象与统一接口，可泛化于多样任务，支持运行时按需生成无限查询。它集成网络模拟器，在执行中测量正确性、安全性与延迟。我们在三个典型应用上验证发现：(1) NetArena显著提升统计可靠性，置信区间重叠率从85%降至0；(2) 代理在大规模真实查询中平均性能仅为13%-38%（最低3%）；(3) 能暴露静态基准忽略的细粒度行为。该框架还可用于SFT和RL微调。代码已开源。
原文摘要 · Abstract (English)
As AI agents expand into high-stakes domains like network system operations, evaluating their real-world reliability becomes increasingly critical. However, existing benchmarks risk contamination due to static design, show high statistical variance from limited dataset size, and fail to reflect the complexity of production environments. We present NetArena, a dynamic benchmark generation framework for network applications. NetArena introduces a novel abstraction and unified interface that generalize across diverse tasks, enabling dynamic benchmarking despite the heterogeneity of network workloads. At runtime, users can generate unlimited queries on demand. NetArena integrates with network emulators to measure correctness, safety, and latency during execution. We demonstrate NetArena on three representative applications and find that (1) NetArena significantly improves statistical reliability across AI agents, reducing confidence-interval overlap from 85% to 0, (2) agents achieve only 13-38% average performance (as low as 3%) for large-scale, realistic queries, and (3) it exposes more fine-grained behaviors that static, correctness-only benchmarks miss. NetArena also enables use cases such as SFT and RL fine-tuning on network system tasks. Code is available at https://github.com/Froot-NetSys/NetArena.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。