用语义感知的Wasserstein距离优化大模型对齐，提升生成质量。
Semantic-aware Wasserstein Policy Regularization for Large Language Model Alignment
- 基于熵正则Wasserstein距离设计语义感知正则化项
- 在多个数据集上超越KL与f-散度基线，提升对齐效果
- 适合需要高质量生成结果的对话与内容创作场景
大型语言模型通常通过人类反馈强化学习（RLHF）对齐人类偏好。传统方法使用Kullback-Leibler（KL）散度对参考策略进行正则化，但其仅比较相同位置的词元概率，无法捕捉语义相似性。本文提出基于熵正则化Wasserstein距离的语义感知策略正则化（WPR），引入词元空间的几何结构。该距离的对偶形式将正则化转化为通过最优对偶变量施加于奖励的惩罚项，得到可与标准强化学习算法兼容的可训练目标。实验证明，该方法在多个基准上优于基于KL和f-散度的基线，验证了语义感知策略距离在对齐任务中的优势。代码已开源：https://github.com/aailab-kaist/WPR。
原文摘要 · Abstract (English)
Large language models (LLMs) are commonly aligned with human preferences using reinforcement learning from human feedback (RLHF). In this method, LLM policies are generally optimized through reward maximization with Kullback-Leibler (KL) divergence regularization of the reference policy. However, KL and its $f$-divergence variants only compare token probabilities at identical indices, failing to capture semantic similarity. We propose Wasserstein Policy Regularization (WPR), a semantic-aware regularization for the RLHF framework based on the entropy-regularized Wasserstein distance, which incorporates the geometry of the token space. The dual formulation of the distance expresses the regularization as penalty terms applied to the reward via optimal dual variables, which yield a tractable objective compatible with standard RL algorithms. Empirically, our method outperforms KL- and $f$-divergence-based baselines, demonstrating the benefits of semantic-aware policy distances for alignment. Our code is available at https://github.com/aailab-kaist/WPR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。