为代码生成设计新强化学习方法，解决测试时无标签训练难题。
Entropy-Regularized Rank-Masked Policy Optimization for Test-Time Reinforcement Learning in Code Generation
- 用探针输入和行为一致性定义奖励信号
- 提升代码通过率，零样本迁移效果显著
- 适合需要自适应优化的代码生成场景
现有测试时强化学习（TTRL）方法依赖答案级自投票获取奖励，但在代码生成中因程序无法通过表面形式比较而失效。为此，本文提出探针驱动的TTRL，从问题描述构建无输出探针输入，执行候选程序并基于行为一致性定义探针共识奖励（PCR），为开放词汇程序提供行为训练信号。但PCR并非完全可靠，仍可能被虚假共识欺骗。因此引入熵正则化秩掩码策略优化（ERPO），通过秩掩码将低PCR转为保守负更新，并以熵上限控制策略漂移。在多个编码基准上，ERPO显著提升pass@1和pass@k指标，适用于领域内适应与零样本迁移。
原文摘要 · Abstract (English)
Existing methods for test-time reinforcement learning (TTRL) derive rewards from answer-level self-voting on unlabeled test-time tasks with canonical answers, but this breaks down for code generation because programs cannot be compared by surface form and therefore do not directly provide a usable training signal. To make TTRL applicable to code generation, we propose probe-driven TTRL, which constructs output-free probe inputs from the problem statement, executes candidate programs on these probes, and defines a Probe Consensus Reward (PCR) from the resulting behavioral agreement. PCR provides a behavioral training signal for open-vocabulary programs, but it is not a fully reliable verifier and remains susceptible to reward hacking through spurious consensus. We therefore introduce Entropy-Regularized Rank-Masked Policy Optimization (ERPO), which converts low PCR into conservative negative updates through rank masking and controls policy drift with an entropy ceiling. On coding benchmarks, ERPO substantially improves pass@1 and pass@k in both in-domain adaptation and zero-shot transfer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。