AIChilles自动发现AI生成系统中的隐藏缺陷，提升可靠性。
AIChilles: Automatically Uncovering Hidden Weaknesses in AI-Evolved Systems
- 结合参数提取与约束推理，智能生成测试负载。
- 在30个AI演化程序中发现49处隐蔽缺陷。
- 适合关注AI系统安全性的开发者和测试团队。
计算机系统领域近年兴起基于AI的系统演化，AI代理可迭代重写系统。如AdaEvolve和Engram等框架报告相较人工设计算法有12%-60%的性能提升。然而，这些AI演化程序在未见工作负载下可能表现更差，或出现可扩展性退化。鉴于AI生成代码的速度与规模，亟需自动化机制来揭示此类隐藏弱点。为此，我们提出AIChilles，输入基线程序P与AI演化程序P'，该工具搜索导致P'在正确性、运行时间、内存使用或输出质量上劣于P的有效工作负载。为应对系统应用多样性、弱点类型与潜在错误的复杂性，AIChilles融合确定性工作负载参数提取、基于代理的约束推断、差异型断言及代码频率覆盖率，以发现多样化故障。在五个系统应用与30个AI演化程序中，共发现49种不同类型的隐藏弱点。此外，我们验证将AIChilles纳入AI驱动开发流程可有效缓解部分缺陷。
原文摘要 · Abstract (English)
The computer systems community has recently seen growing interest in AI-driven system evolution, where AI agents iteratively rewrite systems. Frameworks such as AdaEvolve and Engram report 12-60% score improvements over human-designed algorithms. While these results are promising, there are practical concerns if these AI-evolved programs can perform worse on unseen workloads and exhibit scalability regressions. Given the speed and scale of AI-generated code, we need automated mechanisms to uncover such identify hidden weaknesses in AI-evolved systems programs. To this end, we develop AIChilles that takes as input a baseline program $P$ and an AI-evolved program $P'$, AIChilles searches for valid workloads where $P'$ regresses relative to $P$ in correctness, runtime, memory usage, or output quality. To tackle the diversity in system applications, weakness types and potential bugs, AIChilles combines deterministic workload-parameter extraction, agent-based constraint inference, differential oracles, and code-frequency coverage to discover diverse failures. Across five system applications and 30 AI-evolved programs, AIChilles finds 49 distinct hidden weaknesses. We also show that explicitly including AIChilles in the AI-driven development lifecycle can mitigate several of these weaknesses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。