用大模型提升新语言MOJO的模糊测试，发现13个未知漏洞。
LLMs are All You Need? Improving Fuzz Testing for MOJO with Large Language Models
- 设计自适应框架，动态优化提示词并过滤无效代码。
- 测试有效性提升，覆盖更多API，检测出13个未发现的漏洞。
- 为新兴编程语言的AI测试提供可复用的新方法，适合语言研发者。
大语言模型（LLMs）的快速发展推动了软件测试的自动化，尤其在模糊测试中能生成多样且高效的测试输入，显著提升软件可靠性。与此同时，MOJO作为一种融合Python易用性与C/C++高效性的高性能人工智能编程语言，为提升AI模型可扩展性和可编程性带来新机遇。然而，作为新兴语言，MOJO缺乏完善的测试框架和足够的训练语料，导致大模型易产生语法正确但语义错误的代码，严重削弱模糊测试效果。为此，本文提出首个专为新兴语言零样本学习环境设计的自适应大模型模糊测试框架MOJOFuzzer。该框架采用多阶段流程，在执行前系统性剔除低质量输入，并根据运行时反馈动态调整大模型提示词，实现迭代优化。实验表明，MOJOFuzzer显著提升了测试用例的有效性、API覆盖率及漏洞检测能力，优于传统模糊测试和现有先进大模型测试方法。利用该框架，我们首次对MOJO进行了大规模模糊测试，发现了13个此前未知的漏洞。本研究不仅推进了大模型驱动的软件测试发展，也为新兴编程语言的测试提供了基础方法论。
原文摘要 · Abstract (English)
The rapid development of large language models (LLMs) has revolutionized software testing, particularly fuzz testing, by automating the generation of diverse and effective test inputs. This advancement holds great promise for improving software reliability. Meanwhile, the introduction of MOJO, a high-performance AI programming language blending Python's usability with the efficiency of C and C++, presents new opportunities to enhance AI model scalability and programmability. However, as a new language, MOJO lacks comprehensive testing frameworks and a sufficient corpus for LLM-based testing, which exacerbates model hallucination. In this case, LLMs will generate syntactically valid but semantically incorrect code, significantly reducing the effectiveness of fuzz testing. To address this challenge, we propose MOJOFuzzer, the first adaptive LLM-based fuzzing framework designed for zero-shot learning environments of emerging programming languages. MOJOFuzzer integrates a mutil-phase framework that systematically eliminates low-quality generated inputs before execution, significantly improving test case validity. Furthermore, MOJOFuzzer dynamically adapts LLM prompts based on runtime feedback for test case mutation, enabling an iterative learning process that continuously enhances fuzzing efficiency and bug detection performance. Our experimental results demonstrate that MOJOFuzzer significantly enhances test validity, API coverage, and bug detection performance, outperforming traditional fuzz testing and state-of-the-art LLM-based fuzzing approaches. Using MOJOFuzzer, we have conducted a first large-scale fuzz testing evaluation of MOJO, uncorvering 13 previous unknown bugs. This study not only advances the field of LLM-driven software testing but also establishes a foundational methodology for leveraging LLMs in the testing of emerging programming languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。