用大模型从自然语言生成可验证的测试语法，提升代码竞赛测试用例质量。
SAGE:Specification-Aware Grammar Extraction for Automated Test Case Generation with LLMs
- 基于少量标注样本和强化学习，将规格描述转为带计数器的上下文无关语法
- 在17个大模型中表现最优，语法有效率提升15.92%，测试效果提升12.34%
- 适合自动化测试生成、编程竞赛评测与低资源语法提取场景
基于语法的测试用例生成在编程竞赛中已证明有效，但如何从自然语言规格中生成有效且通用的语法仍是关键挑战，尤其在监督数据有限的情况下。上下文无关语法带计数器（CCFG）通过在推导过程中存储和重用计数器值，可形式化表达带逻辑约束的规格。本文探索使用开源大语言模型（LLMs）在少量标注样本和可验证奖励引导的强化学习下，自动提取CCFG。首先微调开源LLM实现规格到语法的翻译，再应用组相对策略优化（GRPO）提升语法的有效性与泛化能力。同时研究了迭代反馈对开源与闭源模型修正语法错误的效果。实验表明，所提方法SAGE在语法质量与测试有效性上均优于17个开源与闭源模型，语法有效率提升15.92个百分点，测试效果提升12.34个百分点。代码与数据集已匿名公开于：https://anonymous.4open.science/r/SAGE-5714
原文摘要 · Abstract (English)
Grammar-based test case generation has proven effective for competitive programming problems, but generating valid and general grammars from natural language specifications remains a key challenge, especially under limited supervision. Context-Free Grammars with Counters (CCFGs) have recently been introduced as a formalism to represent such specifications with logical constraints by storing and reusing counter values during derivation. In this work, we explore the use of open-source large language models (LLMs) to induce CCFGs from specifications using a small number of labeled examples and verifiable reward-guided reinforcement learning. Our approach first fine-tunes an open-source LLM to perform specification-to-grammar translation, and further applies Group Relative Policy Optimization (GRPO) to enhance grammar validity and generality. We also examine the effectiveness of iterative feedback for open and closed-source LLMs in correcting syntactic and semantic errors in generated grammars. Experimental results show that our approach SAGE achieves stronger generalization and outperforms 17 open and closed-source LLMs in both grammar quality and test effectiveness, improving over the state-of-the-art by 15.92%p in grammar validity and 12.34%p in test effectiveness. We provide our implementation and dataset at the following anonymous repository:https://anonymous.4open.science/r/SAGE-5714
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。