让大模型学会自我反思，提升对齐效果
InSPO: Unlocking Intrinsic Self-Reflection for LLM Preference Optimization
- 基于上下文和对比响应联合优化策略
- 在多个数据集上胜率提升，长度控制更优
- 无需修改模型结构，适合实际部署
直接偏好优化（DPO）及其变体因其简洁性和离线稳定性已成为大语言模型对齐的标准方法。然而，我们识别出两个根本性局限：其一，最优策略依赖于任意的建模选择（如标量化函数、参考策略），导致行为反映参数化伪影而非真实偏好；其二，孤立地处理生成响应，未能利用成对数据中的比较信息，使模型内在自省能力未被释放。为此，我们提出内在自反思偏好优化（InSPO），推导出同时依赖上下文与备选响应的全局最优策略。理论证明该方法优于DPO/RLHF，且对标量化和参考策略选择具有不变性。InSPO可即插即用，无需架构改动或推理开销。实验显示其在胜率与长度控制指标上持续提升，验证了激发自省能力可带来更鲁棒、更符合人类偏好的大模型。代码已开源：https://github.com/Skylanding/InSPO。
原文摘要 · Abstract (English)
Direct Preference Optimization (DPO) and its variants have become standard for aligning Large Language Models due to their simplicity and offline stability. However, we identify two fundamental limitations. First, the optimal policy depends on arbitrary modeling choices (scalarization function, reference policy), yielding behavior reflecting parameterization artifacts rather than true preferences. Second, treating response generation in isolation fails to leverage comparative information in pairwise data, leaving the model's capacity for intrinsic self-reflection untapped. To address it, we propose Intrinsic Self-reflective Preference Optimization (InSPO), deriving a globally optimal policy conditioning on both context and alternative responses. We prove this formulation superior to DPO/RLHF while guaranteeing invariance to scalarization and reference choices. InSPO serves as a plug-and-play enhancement without architectural changes or inference overhead. Experiments demonstrate consistent improvements in win rates and length-controlled metrics, validating that unlocking self-reflection yields more robust, human-aligned LLMs. Our Code is available at https://github.com/Skylanding/InSPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。