用联邦学习让大模型对齐多元人类偏好，既快又隐私。
PluralLLM: Pluralistic Alignment in LLMs via Federated Learning
- 通过联邦平均聚合多方偏好更新，避免数据集中收集。
- 收敛速度提升46%，对齐得分提高4%，公平性接近中心化训练。
- 适合关注隐私保护与多群体公平对齐的研究者或应用方。
确保大语言模型（LLMs）与多样化的人类偏好对齐，同时保护隐私和实现公平，仍是挑战。现有方法如基于人类反馈的强化学习（RLHF）依赖集中式数据收集，计算成本高且侵犯隐私。我们提出PluralLLM，一种基于联邦学习的方法，使多个用户群体可协作训练基于Transformer的偏好预测器，无需共享敏感数据，该模型亦可作为奖励模型用于对齐LLMs。方法采用联邦平均（FedAvg）高效聚合偏好更新，在问答偏好对齐任务上，实现46%更快的收敛速度，对齐分数提升4%，组公平性几乎与集中式训练相当。结果表明，联邦偏好学习为对齐多样化人类价值观提供了可扩展且隐私保护的替代方案。
原文摘要 · Abstract (English)
Ensuring Large Language Models (LLMs) align with diverse human preferences while preserving privacy and fairness remains a challenge. Existing methods, such as Reinforcement Learning from Human Feedback (RLHF), rely on centralized data collection, making them computationally expensive and privacy-invasive. We introduce PluralLLM a federated learning-based approach that enables multiple user groups to collaboratively train a transformer-based preference predictor without sharing sensitive data, which can also serve as a reward model for aligning LLMs. Our method leverages Federated Averaging (FedAvg) to aggregate preference updates efficiently, achieving 46% faster convergence, a 4% improvement in alignment scores, and nearly the same group fairness measure as in centralized training. Evaluated on a Q/A preference alignment task, PluralLLM demonstrates that federated preference learning offers a scalable and privacy-preserving alternative for aligning LLMs with diverse human values.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。