通过混合语义邻近嵌入提升策略优化，解决数学推理中路径冗余与语义不稳问题。
N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimization
- 用锚点词及其语义邻近词的嵌入混合生成新输入，保持语义连贯性
- 在DeepSeek-R1-Distill-Qwen模型上显著优于基线，在分布外任务上仍表现稳健
- 适合需要高质量多样化推理路径的数学智能体开发
大型语言模型在数学推理中的成功高度依赖于推理过程中生成多样且有效的解题路径。然而，现有采样方法面临根本矛盾：基于词元的采样常导致仅表达方式不同的重复路径，而基于随机噪声的嵌入级方法则易破坏语义一致性。为此，我们提出N-GRPO，一种融入组相对策略优化（GRPO）框架的新探索策略。该方法不依赖词元级采样或原始嵌入级噪声，而是采用语义邻近混合机制，通过混合锚定词与其最近语义邻近词的嵌入动态构建输入表示，从而在严格遵循局部语义流形的前提下注入多样性。在不同规模的DeepSeek-R1-Distill-Qwen模型上的实验表明，N-GRPO不仅在数学推理基准上持续超越强基线，还在分布外任务上展现出鲁棒的泛化能力。
原文摘要 · Abstract (English)
The success of Large Language Models in mathematical reasoning relies heavily on the generation of diverse and valid solution paths during the rollout phase. However, current rollout techniques face a fundamental trade-off: token-level sampling often yields redundant trajectories that differ only in rephrasing, while embedding-level methods utilizing random noise frequently disrupt semantic consistency. To resolve this, we introduce N-GRPO, a novel exploration strategy integrated into the Group Relative Policy Optimization (GRPO) framework. Rather than relying on token-level sampling or native embedding-level noise, our approach leverages Semantic Neighbor Mixing. This mechanism dynamically constructs input representations by mixing the embeddings of an anchor token and its nearest semantic neighbors, thereby injecting diversity while strictly adhering to the local semantic manifold. Experimental evaluations on the DeepSeek-R1-Distill-Qwen models across different sizes show that N-GRPO not only achieves consistent improvements over strong baselines on math reasoning benchmarks but also exhibits robust generalization capabilities on out-of-distribution tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。