用智能代理系统自动检测JavaScript解释器差异，减少人工校验错误。
SmartOracle -- An Agentic Approach to Mitigate Noise in Differential Oracles
- 拆分人工验证流程为多个LLM专用代理协同决策
- 召回率达0.84，误报率18%，分析速度提升4倍
- 适合自动化测试团队与语言引擎开发者使用
差异模糊测试通过在相同规范的不同实现间执行相同输入来发现缺陷，如JavaScript解释器。输出验证需依赖人工构建的判别器，成本高、易出错，且规范更新后需重复劳动。本文提出SmartOracle，将人工甄别流程分解为多个大型语言模型子代理，通过整合终端运行证据与定向规范查询结果做出最终判断。在历史基准测试中，其召回率为0.84，误报率为18%。相较于串行的Gemini 2.5 Pro基线，准确率更高，分析时间缩短4倍，API成本降低10倍。在实际模糊测试活动中，成功识别并报告了V8、JavaScriptCore和GraalJS等主流引擎中此前未知的规范级问题。SmartOracle在JavaScript领域的成功表明该智能体架构可推广至其他软件系统，未来工作将探索此方向。
原文摘要 · Abstract (English)
Differential fuzzers detect bugs by executing identical inputs across distinct implementations of the same specification, such as JavaScript interpreters. Validating the outputs requires an oracle and for differential testing of JavaScript, these are constructed manually, making them expensive, time-consuming, and prone to false positives. Worse, when the specification evolves, this manual effort must be repeated. Inspired by the success of agentic systems in other SE domains, this paper introduces SmartOracle. SmartOracle decomposes the manual triage workflow into specialized Large Language Model (LLM) sub-agents. These agents synthesize independently gathered evidence from terminal runs and targeted specification queries to reach a final verdict. For historical benchmarks, SmartOracle achieves 0.84 recall with an 18% false positive rate. Compared to a sequential Gemini 2.5 Pro baseline, it improves triage accuracy while reducing analysis time by 4$\times$ and API costs by 10$\times$. In active fuzzing campaigns, SmartOracle successfully identified and reported previously unknown specification-level issues across major engines, including bugs in V8, JavaScriptCore, and GraalJS. The success of SmartOracle's agentic architecture on Javascript suggests it might be useful other software systems- a research direction we will explore in future work.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。