提出新评估方法，量化合成验证对代码与推理能力的提升效果
Scoring Verifiers: Evaluating Synthetic Verification for Code and Reasoning
- 将现有代码评测集改造为评分排序数据集，支持合成验证器评估
- 发现推理能力显著提升用例生成质量，用例数量越多验证越准
- 适合关注LLM代码生成与强化学习验证的研究者参考
合成验证技术如自动生成测试用例和奖励建模，是提升大语言模型（LLM）编码能力的重要手段。近期，代码验证在通过强化学习增强LLM推理能力方面取得显著成效。本文提出一种方法，可将现有编码基准转换为评分与排序数据集，用于评估合成验证器的有效性，并设计多种指标衡量不同维度的性能。基于该方法，我们发布了四个新基准（HE-R、HE-R+、MBPP-R、MBPP-R+），并使用标准、基于推理和基于奖励的LLM分析了合成验证方法。实验表明，推理能力能显著提升测试用例生成质量，增加测试用例数量可有效提高验证准确率。
原文摘要 · Abstract (English)
Synthetic verification techniques such as generating test cases and reward modelling are common ways to enhance the coding capabilities of large language models (LLM) beyond predefined tests. Additionally, code verification has recently found great success as a critical component in improving reasoning capability of LLMs via reinforcement learning. In this paper, we propose an approach which can transform existing coding benchmarks into scoring and ranking datasets to evaluate the effectiveness of synthetic verifiers. We also propose multiple metrics to measure different aspects of the synthetic verifiers with the proposed benchmarks. By employing the proposed approach, we release four new benchmarks (HE-R, HE-R+, MBPP-R, and MBPP-R+), and analyzed synthetic verification methods with standard, reasoning-based, and reward-based LLMs. Our experiments show that reasoning can significantly improve test case generation and that scaling the number of test cases enhances the verification accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。