通过配对感知重加权,让强化学习推理更省算力且更准确。
PAIR: Pairwise-Aware Inclusion Reweighting for Adaptive Rollout Allocation in RLVR

- 构建对比图模型,用配对梯度修正传统方法的偏差。
- 在相同算力下,准确率提升1.2~1.4,少用50%以上生成token。
- 适合追求高效高精度推理的RLVR研究者与工程落地者。
基于可验证奖励的强化学习(RLVR)主要消耗算力在生成长推理轨迹组上。现有分配器根据单个样本的难易度或效用分配预算,但存在统计偏差:未截断的留一法组相对评分梯度并非独立项之和,而是成对滚动生成的二阶U统计量。完成一个轨迹会揭示其与其他轨迹的对比信息,自适应终点选择影响可观测配对项。本文提出PAIR(配对感知包含重加权),将短轨迹前缀视为顶点,配对梯度项视为边,构建对比图。仅基于前缀的预测器估计正确性与剩余词元成本;凸优化设计在预期后缀词元预算下选择正向继续概率;已完成顶点诱导的每条边按其记录的联合包含概率反向加权。在条件独立的策略内滚动生成与未标准化目标下,该估计器对完整候选对梯度为设计无偏。在Qwen3-1.7B/4B上的计算匹配实验表明,PAIR相比最强点对点分配器平均准确率分别提升+1.2和+1.4,同时生成词元数较全组GRPO减少51%和52%。冻结种群估计器审计确认,未加权自适应选择存在偏差,而配对包含校正能恢复等价于完整对的期望目标,且在匹配后缀成本下表现一致。
原文摘要 · Abstract (English)
Reinforcement learning with verifiable rewards (RLVR) spends most of its compute generating groups of long reasoning trajectories. Recent allocators reduce this cost by assigning budgets to prompts, rollouts, or tokens according to a pointwise notion of difficulty or utility. We identify a statistical mismatch: the unclipped leave-one-out group-relative score gradient is not a sum of independent point contributions, but a second-order U-statistic over pairs of rollouts. Completing one rollout therefore reveals contrast with every other completed rollout, and adaptive endpoint selection changes which pair terms are observable. We introduce PAIR (Pairwise-Aware Inclusion Reweighting), which treats short rollout prefixes as vertices and pair-gradient terms as edges of a contrast graph. A prefix-only predictor estimates correctness and remaining token cost; a convex design chooses positive continuation probabilities under an expected suffix-token budget; and each edge induced by completed vertices is inverse-weighted by its logged joint inclusion probability. Under conditionally independent on-policy rollouts and an unclipped, unstandardized objective, the resulting estimator is design-unbiased for the complete candidate-pair gradient. Across compute-matched RLVR runs on Qwen3-1.7B/4B, PAIR improves average accuracy by +1.2 and +1.4 over the strongest pointwise allocator while using 51% and 52% fewer generated tokens than full-group GRPO. A frozen-population estimator audit confirms that unweighted adaptive selection is biased, whereas pair-inclusion correction recovers the complete-pair target at matched suffix cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。