解决联邦学习中用户偏好冲突问题，实现个性化推荐。
Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences
- 用可动态更新的混合先验稳定分布式推断
- 在HH-RLHF数据集上实现偏好解耦与切换
- 适合需要隐私保护的个性化模型场景
联邦学习为对齐大语言模型提供了一条保护隐私的路径；然而，现有框架通常采用单一奖励模型，不可避免地平均了内在冲突的用户偏好（如有用性与无害性）。虽然变分偏好学习（VPL）提供了个性化的可能，但将其应用于去中心化环境面临根本挑战：由本地数据稀缺和异质性引发的后验坍缩。本文提出联邦变分偏好对齐与Gumbel-Softmax先验框架（FedVPA-GP），旨在不牺牲隐私的前提下解耦多样偏好。为稳定变分推断，我们引入联邦混合先验，使客户端能利用聚合的群体分布作为动态先验。此外，引入正交损失，显式强制潜空间中偏好原型的分离。在HH-RLHF数据集上的实验表明，FedVPA-GP显著优于单体基线，成功解耦冲突用户意图并支持动态偏好切换。
原文摘要 · Abstract (English)
Federated Learning (FL) offers a privacy-preserving pathway for aligning Large Language Models (LLMs); however, existing frameworks typically enforce a monolithic reward model, inevitably averaging out inherently conflicting user preferences (e.g., helpfulness vs. harmlessness). While Variational Preference Learning (VPL) offers a pathway to personalization, adapting it to decentralized settings presents a fundamental challenge: posterior collapse driven by severe local data scarcity and heterogeneity. In this paper, we propose Federated Variational Preference Alignment with Gumbel-Softmax Prior (FedVPA-GP), a framework designed to disentangle diverse preferences without compromising privacy. To stabilize variational inference, we introduce a Federated Mixture Prior that enables clients to leverage the aggregate population distribution as a dynamic prior. Furthermore, we incorporate an Orthogonal Loss that explicitly enforces the separation of preference prototypes in the latent space. Experiments on the HH-RLHF dataset demonstrate that FedVPA-GP significantly outperforms monolithic baselines, successfully disentangling conflicting user intents and enabling dynamic preference switching.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。