用大模型生成微服务测试用例,发现提示词比模型大小更重要。
LLM-Based Robustness Testing of Microservice Applications: An Empirical Study
- 用提示词工程引导大模型生成多样化测试用例
- 单一模型+最优提示词覆盖全部缺陷模式
- 引入缺陷变异分类体系提升测试效果
微服务API中的畸形、缺失或边界值输入可能引发级联故障,威胁系统可靠性。鲁棒性测试通过系统化输入来暴露服务器端问题,但生成多样且有效的测试用例仍具挑战。大语言模型可从API规范生成此类测试,但不同模型与提示策略是否产生多样缺陷集尚不明确。本研究对3个开源LLM(14B-70B参数)采用7种提示策略,针对两个架构不同的微服务系统(一个Java单语言系统:6服务,9种缺陷;一个多语言系统:27服务,14种缺陷)进行受控实验,共获得38次有效运行和663个生成测试。结果表明,提示策略解释的多样性变异远超模型规模:结构化提示导致多样性完全丧失;单一模型在三种提示策略下即可在其中一个系统实现完整缺陷模式覆盖,优于固定提示下的多模型集成。提出两种新策略——Guided与GuidedFewShot,将已有鲁棒性测试研究中的变异分类体系作为领域上下文嵌入。GuidedFewShot在两个系统中均达到最高单次运行覆盖率(分别覆盖5/9和8/14种缺陷模式),且跨模型相似性低。关键发现:仅依赖分类规则不足,大模型无法区分关键字段缺失与值为空的变异,需具体示例辅助。结论在两系统间一致复现。
原文摘要 · Abstract (English)
Malformed, missing, or boundary-value inputs in microservice APIs can cascade across dependent services, threatening reliability. Robustness testing systematically exercises such inputs to expose server-side failures, but generating diverse, effective tests remains challenging. Large Language Models can generate such tests from API specifications; however, it is unknown whether different models and prompt strategies produce diverse failure sets or converge on the same failures. We report a controlled experiment applying 7 prompt strategies to 3 open-source LLMs (14B-70B parameters) targeting 2 architecturally distinct microservice systems: one Java monolingual (6 services, 9 failure modes) and one polyglot (27 services, 14 failure modes), yielding 38 valid runs and 663 generated tests. We find that prompt strategy explains more variation in diversity than model size: a Structured prompt collapses diversity entirely, while a single model varied across three prompt strategies achieves complete failure-mode coverage on one system, outperforming any multi-model ensemble under a fixed prompt. We introduce two strategies, Guided and GuidedFewShot, that embed a mutation taxonomy from prior robustness testing research as domain context. GuidedFewShot achieves the highest single-run coverage on both systems (5 of 9 and 8 of 14 failure modes) while maintaining low cross-model similarity. A key lesson is that taxonomy rules alone are insufficient: LLMs cannot distinguish key-absent from value-empty mutations without concrete examples. Findings replicate across both systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。