用响应内互信息动态调节偏好优化，省去调参且提速15%以上
Intrinsic Mutual Information as a Modulator for Preference Optimization
- 基于响应级互信息自动调节偏好贡献，无需人工调参
- 在多个数据集上优于现有方法，训练开销降低超15%
- 轻量设计适合快速部署，特别适合资源有限的场景
离线偏好优化方法（如直接偏好优化，DPO）在对齐大语言模型与人类价值观方面具有显著优势。然而，这些方法通常需要额外的超参数调优，导致大量时间开销。尽管已有研究提出多种改进，但效果仍有限，且未能完全摆脱对超参数调优的依赖。本文提出一种轻量高效的离线偏好优化框架RMiPO，利用响应级内在互信息实现偏好优化中的超参数调制，以极低计算成本动态解耦偏好贡献。大量实验表明，RMiPO在多个基准上持续优于现有方法，同时将训练开销减少超过15%。代码已开源：https://github.com/liavonpenn/rmipo。
原文摘要 · Abstract (English)
Offline preference optimization methods, such as Direct Preference Optimization (DPO), offer significant advantages in aligning Large Language Models (LLMs) with human values. However, achieving optimal performance with these methods typically involves additional hyperparameter tuning, resulting in substantial time overhead. Although prior work has proposed a range of improvements, these methods remain limited in effectiveness and have not fully eliminated reliance on hyperparameter tuning. In this work, we propose RMiPO, a lightweight and efficient framework for offline preference optimization. RMiPO leverages intrinsic Response-level Mutual information for Preference Optimization with hyperparameter modulation, dynamically decoupling preference contributions at negligible additional computational cost. Extensive experimental results demonstrate that RMiPO achieves consistently superior performance over existing methods while reducing training overhead by more than 15\%. Our code is available at https://github.com/liavonpenn/rmipo.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。