用强化学习生成更少、更高效的单元测试,一次搞定漏洞检测与测试精简。
Ockhamareto: Pareto-Gated Segment-Level Credit Assignment for Concise Unit-Test Generation with Reinforcement Learning

- 基于帕累托最优和奥卡姆剃刀原则,只奖励不被支配的测试生成路径。
- 在5个测试用例下捕获49.9%缺陷,比最强基线多18.6个百分点,测试数减少40%。
- 适用于追求高效测试生成的开发者,尤其适合大模型规模下的代码验证场景。
我们提出Ockhamareto,一种基于奥卡姆剃刀与帕累托最优原则的单次强化学习框架,用于单元测试生成与选择。其核心包含:(i) 帕累托门控奖励机制,仅奖励在(变异体数量,测试数)空间中非被支配的轨迹;(ii) 令牌级段落信用分配,将每个测试的变异体杀灭效果归因于其单元测试块中的具体令牌。在UnLeakedTestBench(ULT)上,Ockhamareto严格帕累托主导最强基线MIST-RL。在N=5时,其变异体得分达49.9%,优于基线31.3%;平均测试数仅2.60,远低于基线4.67,每测试效率提升3.4倍。该优势在四个基准(HumanEval+、MBPP+、CodeContests、TestGenEval-Lite)上全面显现,始终以最小测试集实现最高变异体覆盖率与代码覆盖率。在4B、9B、27B模型规模下,均比当前最优提升+30至+35个百分点。研究还发现,帕累托前沿上的最优权衡点与函数大小等易计算代理指标无关,表明必须通过帕累托分析识别实际工程权衡。
原文摘要 · Abstract (English)
We introduce \textbf{Ockhamareto}, a single-shot GRPO framework for unit-test generation and selection, based on the principles of \emph{Ockham's Razor} and \emph{Pareto Optimality}. Ockhamareto has two principal components: (i)~a \emph{Pareto-gated Bonus} that rewards only rollouts non-dominated in~(mutation, $-$\#tests) space, and (ii)~\emph{Token-level Segment Credit}, which attributes each test's marginal mutation kills back to the tokens of its unit-test block. On the \emph{UnLeakedTestBench~(ULT)}, Ockhamareto \emph{strictly Pareto-dominates} the strongest RL baseline~(\emph{MIST-RL}). Furthermore, it dominates on {\em each and all} optimization objectives, catching more bugs ($49.9\%$ vs $31.3\%$ mutation score at $N{=}5$), using \emph{fewer} tests ($2.60$ vs $4.67$ on average), thereby achieving $3.4\times$ the per-test trade-off improvement. The advantage is found in all four benchmarks~(\emph{HumanEval+}, \emph{MBPP+}, \emph{CodeContests}, \emph{TestGenEval-Lite}): Ockhamareto leads both mutation and coverage metrics on every one, always with the smallest suite. Ockhamareto also outperforms the state-of-the-art at all model scales, adding $+30$--$35$~pp mutation at 4B, 9B, and 27B model sizes. We also show that the knee point of the optimal trade-off between efficiency and effectiveness on the Pareto front is not correlated with obvious more easily computed proxy metrics, such as function size. This finding motivates the Pareto front computation; it is needed to identify this crucial engineering trade-off for each function under test.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。