用最短对偶响应提升推理强化学习效率，更简洁准确
BPPO: Binary Prefix Policy Optimization for Efficient GRPO-Style Reasoning RL with Concise Responses
- 仅用最短正确与错误回应作为更新单元，减少冗余计算
- 速度提升6.08倍，响应长度减少30-50%，精度不降
- 适合追求高效、简洁推理模型的开发者与研究者
群体相对策略优化（GRPO）广泛用于训练推理模型，但每次更新所有采样完成项带来高昂开销，并可能强化冗长推理路径。本文通过梯度相似性分析发现，在同一提示组内，同类完成项常产生高度相似的更新方向，而正确-错误对则提供更显著的对比信号。受此启发，我们提出二元前缀策略优化（BPPO），以最短正确完成项和最短错误完成项构成紧凑更新单元，同时保留全组优势归一化。BPPO进一步通过自适应完成项调度和前缀聚焦优化提升效率：仅更新响应前缀，避免强化冗余后缀，鼓励生成更简洁回答。在GSM8K、MATH和Geo3K上的实验表明，BPPO相比GRPO实现最高6.08倍提速，保持竞争力准确率，且无需显式长度惩罚，平均响应长度减少约30%-50%。
原文摘要 · Abstract (English)
Group Relative Policy Optimization (GRPO) is widely used for training reasoning models, but updating all sampled completions in each group incurs substantial cost and can reinforce verbose reasoning trajectories. In this paper, we study whether all completions provide equally useful update signals in GRPO-style reasoning RL. Our gradient-similarity analysis shows that, within the same prompt group, same-class completions often induce highly similar update directions, whereas correct-incorrect pairs provide more distinct contrastive signals. Motivated by this observation, we propose Binary Prefix Policy Optimization (BPPO), which uses the shortest correct completion and the shortest incorrect completion as a compact update unit while preserving full-group advantage normalization. BPPO further improves efficiency with adaptive completion scheduling and prefix-focused optimization; by updating only response prefixes, it avoids reinforcing redundant suffixes and encourages more concise responses. Experiments on GSM8K, MATH, and Geo3K show that BPPO achieves up to 6.08x speedup over GRPO while maintaining competitive accuracy, and reduces mean response length by approximately 30-50% without modifying the reward with an explicit length penalty.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。