无需公开测试用例,大模型也能自动生成正确代码。
You Don't Need Public Tests to Generate Correct Code
- 让大模型自主生成测试输入并模拟执行,实现自我纠错。
- 在LiveCodeBench v6上性能媲美依赖测试的顶尖方法。
- 适合无公开测试的真实开发场景,降低人工成本。
多智能体系统常用于自主代码生成,在复杂算法问题求解中表现优异。现有方法通过仿真引导的规划与调试来提升程序正确性,依赖人工编写的公开测试用例作为仿真与调试的锚点。然而,手写完整的输入输出对消耗大量人力,且真实场景中往往无法提前获取真值样例,导致现有方法仅适用于竞赛类数据集。我们还发现,依赖公开测试会产生“过度自信缺口”,使模型过拟合基础样例,而在隐藏测试集上表现不佳。相反,我们指出外部输入并非必要。大模型具备自主构造有效输入并模拟执行流的能力。基于此,提出DryRUN框架:让大模型迭代规划、自动生成测试输入、进行模拟执行,从而缓解算法过度自信。在LiveCodeBench v6(2025年3月后版本)上的评估显示,DryRUN在不使用任何公开测试或外部执行信号的前提下,性能可媲美当前最先进的依赖测试的方法,同时减少输出令牌总数。
原文摘要 · Abstract (English)
Multi-agent systems are frequently employed for autonomous code generation, demonstrating strong utility in complex algorithmic problem-solving. Recent studies tackle the difficulty of producing functionally correct programs by leveraging simulation-guided planning and debugging, wherein language models step through execution traces to validate logic. Nevertheless, these methods rely heavily on human-authored public test cases to anchor the simulation and debugging cycles. Hand-crafting exhaustive input-output pairs creates a significant, labor-intensive bottleneck within the software development lifecycle. Since ground-truth examples are seldom accessible before actual implementation in real-world scenarios, this reliance limits existing approaches primarily to curated competitive programming datasets. Additionally, we demonstrate that depending on these public tests creates an "overconfidence gap," leading frameworks to overfit to basic examples and underperform on hidden test suites. Conversely, we note that external input samples are not an absolute requirement for successful code generation. We show that large language models possess the capability to autonomously construct valid inputs and simulate execution flows for self-correction. Building on this, we introduce DryRUN, a framework that removes the necessity for ground-truth data by enabling the LLM to iteratively plan, synthesize its own test inputs, and run simulated executions, thereby mitigating algorithmic overconfidence. Assessments using the LiveCodeBench v6 dataset (post-March 2025) reveal that DryRUN achieves comparable performance to CodeSIM, a state-of-the-art, test-dependent baseline. Notably, it does so entirely without public tests or external execution signals, all while decreasing overall output token usage.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。