提出PEPO算法，无需数据分布知识即可避免偏好学习中的过优化问题。
Provably avoiding over-optimization in Direct Preference Optimization without knowing the data distribution
- 用多个独立子集训练的策略集成实现悲观性，通过最差情况聚合增强鲁棒性。
- 在表格场景下，样本复杂度仅依赖单策略集中系数，优于传统DPO的全策略集中系数。
- 保留DPO简洁性，实测性能优异，适合追求稳定训练的强化学习研究者。
我们提出PEPO（基于悲观集成的偏好优化），一种类似DPO的单步直接偏好优化算法，可在不依赖数据生成分布或显式奖励模型的情况下缓解偏好学习中的过优化问题。PEPO通过在互不重叠的数据子集上训练的策略集成，并采用最差情况聚合方式，使模型更倾向于多模型间的一致性，从而实现悲观性。在表格设置中，PEPO的样本复杂度仅依赖于单策略集中系数，避免了受过优化算法（如DPO）影响的全策略集中系数。理论结果得到实际性能验证，同时保持了DPO风格训练的简洁性和实用性。
原文摘要 · Abstract (English)
We introduce PEPO (Pessimistic Ensemble based Preference Optimization), a single-step Direct Preference Optimization (DPO)-like algorithm to mitigate the well-known over-optimization issue in preference learning without requiring the knowledge of the data-generating distribution or learning an explicit reward model. PEPO achieves pessimism via an ensemble of preference-optimized policies trained on disjoint data subsets and then aggregates them through a worst case construction that favors the agreement across models. In the tabular setting, PEPO achieves sample complexity guarantees depending only on a single-policy concentrability coefficient, thus avoiding the all-policy concentrability which affects the guarantees of algorithms prone to over-optimization, such as DPO. The theoretical findings are corroborated by a convincing practical performance, while retaining the simplicity and the practicality of DPO-style training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。