用希尔伯特空间正交投影优化大模型对齐，避免梯度饱和与熵损失。
Group Orthogonalized Policy Optimization:Group Policy Optimization as Orthogonal Projection in Hilbert Space
- 将策略优化转化为希尔伯特空间中的正交投影，简化约束为线性条件。
- 实测在数学推理任务上性能媲美剪裁方法，且梯度稳定、熵保持不变。
- 无需启发式剪裁，自动实现动作稀疏性，适合高稳定性强化学习场景。
我们提出群正交化策略优化（GOPO），一种基于希尔伯特函数空间几何结构的大语言模型对齐算法。不同于在概率单纯形上优化并继承KL散度的指数曲率，GOPO将对齐过程提升至参考策略π_k下的平方可积函数空间L2(π_k)。在此空间中，单纯形约束退化为线性正交条件⟨v, 1⟩ = 0，定义了一个余维一子空间H₀。通过最小化到无约束目标u_star的距离，得到功耗耗散泛函J(v) = ⟨g, v⟩ - (μ/2)‖v‖²，其最大值由希尔伯特投影定理直接导出。施加边界v ≥ -1后，产生有界希尔伯特投影，诱导精确稀疏性，通过闭式阈值将灾难性差动作的概率置零。为连接理论与实践，GOPO从无限维L2(π_k)投影至由分组采样诱导的有限经验子空间。由于分组归一化优势和为零，强制概率守恒的拉格朗日乘子精确消失，使约束投影简化为无约束经验损失。所得目标具有恒定海森曲率μI、非饱和线性梯度及内在死区机制，无需启发式剪裁。在数学推理基准上的实验表明，GOPO在剪裁类方法趋于平台期的区域仍保持良好泛化能力、稳定的梯度动态与熵保真。
原文摘要 · Abstract (English)
We present Group Orthogonalized Policy Optimization (GOPO), a new alignment algorithm for large language models derived from the geometry of Hilbert function spaces. Instead of optimizing on the probability simplex and inheriting the exponential curvature of Kullback-Leibler divergence, GOPO lifts alignment into the Hilbert space L2(pi_k) of square-integrable functions with respect to the reference policy. Within this space, the simplex constraint reduces to a linear orthogonality condition <v, 1> = 0, defining a codimension-one subspace H0. Minimizing distance to an unconstrained target u_star yields the work-dissipation functional J(v) = <g, v> - (mu / 2) ||v||^2, whose maximizer follows directly from the Hilbert projection theorem. Enforcing the boundary v >= -1 produces a bounded Hilbert projection that induces exact sparsity, assigning zero probability to catastrophically poor actions through a closed-form threshold. To connect this functional theory with practice, GOPO projects from infinite-dimensional L2(pi_k) to a finite empirical subspace induced by group sampling. Because group-normalized advantages sum to zero, the Lagrange multiplier enforcing probability conservation vanishes exactly, reducing the constrained projection to an unconstrained empirical loss. The resulting objective has constant Hessian curvature mu I, non-saturating linear gradients, and an intrinsic dead-zone mechanism without heuristic clipping. Experiments on mathematical reasoning benchmarks show that GOPO achieves competitive generalization while maintaining stable gradient dynamics and entropy preservation in regimes where clipping-based methods plateau.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。