用变分推理解决强化学习中的策略坍缩问题，提升复杂任务表现。
Variational Proximal Policy Optimization
- 基于专家混合架构的粒子变分推断，结合几何控制机制优化策略更新。
- 在Codeforces上提升179 ELO，在AIME上减少32%的令牌消耗。
- 适合关注大模型强化学习与策略优化的研究者参考。
通过近端策略优化实现人类反馈强化学习时常面临策略模式坍缩、探索脆弱和分布漂移等问题。本文提出变分近端策略优化（ extsc{VP}_2 extsc{O}），一种基于粒子的变分推断框架，将策略优化映射为混合专家架构中的史坦变分梯度下降。通过在局部专家原型上使用函数核，并引入专家正交化损失， extsc{VP}_2 extsc{O} 构建了基于几何的近端控制机制，降低对固定裁剪或KL调度的依赖。在33B/4B稀疏混合专家模型上的实验显示，该方法在多个复杂推理基准上取得显著提升：Codeforces任务中获得+179 ELO增益，AIME数学推理任务中令牌消耗减少32%。
原文摘要 · Abstract (English)
Reinforcement Learning from Human Feedback via Proximal Policy Optimization often suffers from policy mode collapse, brittle exploration loops, and distribution drift. This paper introduces Variational Proximal Policy Optimization (\(\textsc{VP}_2\textsc{O}\)), a particle-based variational inference framework that maps policy optimization to Stein Variational Gradient Descent within a Mixture-of-Experts architecture. By leveraging functional kernels over localized expert prototypes alongside an expert orthogonalization loss, \(\textsc{VP}_2\textsc{O}\) introduces a geometry-based proximal-control mechanism that can reduce reliance on fixed clipping or KL schedules. Our results on a 33B/4B sparse Mixture-of-Experts model show several improvements across complex reasoning benchmarks, establishing a \(+\mathbf{179}\) ELO gain on Codeforces and a \(\mathbf{32\%}\) reduction in token count on AIME mathematical reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。