用大模型自动测试通信协议合规性，效率远超传统方法。
iPanda: An LLM-based Agent for Automated Conformance Testing of Communication Protocols
- 基于关键词自动生成全面测试用例，再用推理增强生成可执行代码。
- 在多个协议上测试成功率提升4.675至10.751倍，显著优于纯大模型方案。
- 适合需要高效自动化测试的协议开发与验证团队使用。
一致性测试对确保协议实现符合规范至关重要。然而，传统方法需手动编写大量测试用例和脚本，过程繁琐低效。近年来，大语言模型（LLMs）展现出卓越的文本理解与代码生成能力，为自动化带来新机遇。本文提出iPanda，首个利用大模型实现协议一致性测试自动化的框架。给定协议规范文档与实现代码，iPanda首先采用基于关键词的方法自动生成全面测试用例；随后结合检索增强生成与定制化思维链策略，有效解析实现并生成可执行测试程序；为进一步提升代码质量，引入迭代优化机制，交互式精炼测试脚本；最后通过执行与分析生成测试，系统验证实现与规范的一致性。在多种协议上的综合实验表明，iPanda显著优于纯大模型方法，测试程序生成的成功率（Pass@1）提升4.675至10.751倍。
原文摘要 · Abstract (English)
Conformance testing is essential for ensuring that protocol implementations comply with their specifications. However, traditional testing approaches involve manually creating numerous test cases and scripts, making the process labor-intensive and inefficient. Recently, Large Language Models (LLMs) have demonstrated impressive text comprehension and code generation abilities, providing promising opportunities for automation. In this paper, we propose iPanda, the first framework that leverages LLMs to automate protocol conformance testing. Given a protocol specification document and its implementation, iPanda first employs a keyword-based method to automatically generate comprehensive test cases. Then, it utilizes retrieval-augmented generation and customized CoT strategy to effectively interpret the implementation and produce executable test programs. To further enhance programs' quality, iPanda incorporates an iterative optimization mechanism to refine generated test scripts interactively. Finally, by executing and analyzing the generated tests, iPanda systematically verifies compliance between implementations and protocol specifications. Comprehensive experiments on various protocols show that iPanda significantly outperforms pure LLM-based approaches, improving the success rate (Pass@1) of test-program generation by factors ranging from 4.675 times to 10.751 times.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。