用轻量专家混合模型实现多偏好对齐，提升大模型对齐效果。
PMoL: Parameter Efficient MoE for Preference Mixing of LLM Alignment
- 将低秩适配与专家混合结合，动态融合多偏好。
- 在多个偏好下比基线方法更优，训练成本更低。
- 适合需要多场景对齐的LLM应用开发。
基于人类反馈的强化学习（RLHF）是大型语言模型（LLMs）偏好对齐的有效方法，广泛应用于大模型后训练阶段。然而，传统RLHF难以处理多个竞争性偏好，导致模型与人类偏好对齐度下降。为此，我们提出从模型架构角度出发的偏好混合低秩适配器（PMoL），可适应任意数量偏好的混合。PMoL融合了专家混合（MoE）与低秩适配器（LoRA），首次将该架构用于偏好对齐研究，显著提升性能。通过引入专家组软损失，使MoE具备偏好混合能力。经奖励模型和GPT-4o综合评估，实验结果表明PMoL在偏好混合能力上优于基线方法，且实现更优的偏好对齐，同时训练成本更低。
原文摘要 · Abstract (English)
Reinforcement Learning from Human Feedback (RLHF) has been proven to be an effective method for preference alignment of large language models (LLMs) and is widely used in the post-training process of LLMs. However, RLHF struggles with handling multiple competing preferences. This leads to a decrease in the alignment of LLMs with human preferences. To address this issue, we propose Preference Mixture of LoRAs (PMoL) from the perspective of model architecture, which can adapt to any number of preferences to mix. PMoL combines Mixture of Experts (MoE) and Low Rank Adaptor (LoRA). This architecture is innovatively applied to the research of preference alignment and has achieved significant performance improvement. The expert group soft loss is used to enable MoE with the ability to mix preferences. Through comprehensive evaluation by the reward model and GPT-4o, the experiment results show that PMoL has superior preference mixing capabilities compared to baseline methods. PMoL achieves better preference alignment with lower training costs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。