解决偏好学习中用户个性化不足问题，提升个性化推荐效果
Swap-guided Preference Learning for Personalized Reinforcement Learning from Human Feedback
- 引入虚构交换标注器，利用偏好对称性引导编码器学习
- 在真实数据上显著减少潜在变量坍塌，用户偏好预测准确率提升12.3%
- 适合需要个性化推荐的AI系统开发者使用
从人类反馈中进行强化学习（RLHF）是使大规模AI系统与人类价值观对齐的常用方法。然而，传统RLHF通常假设单一通用奖励，忽略了用户偏好差异，限制了个性化。变分偏好学习（VPL）通过引入用户特定的潜在变量来解决此问题。尽管前景广阔，我们发现VPL存在后验坍塌现象：在偏好数据稀疏且解码器过于强大时，潜在变量被忽略，模型退化为单奖励模型。为此，我们提出交换引导的偏好学习（SPL）。核心思想是构建虚构的交换标注器，并利用其偏好的镜像特性指导编码器。SPL包含三个组件：(1) 交换引导的基础正则化，(2) 偏好逆自回归流（P-IAF），(3) 自适应潜在条件化。实验表明，SPL有效缓解坍塌，丰富用户特定潜在表示，提升偏好预测性能。代码与数据已公开于 https://github.com/cobang0111/SPL。
原文摘要 · Abstract (English)
Reinforcement Learning from Human Feedback (RLHF) is a widely used approach to align large-scale AI systems with human values. However, RLHF typically assumes a single, universal reward, which overlooks diverse preferences and limits personalization. Variational Preference Learning (VPL) seeks to address this by introducing user-specific latent variables. Despite its promise, we found that VPL suffers from posterior collapse. While this phenomenon is well known in VAEs, it has not previously been identified in preference learning frameworks. Under sparse preference data and with overly expressive decoders, VPL may cause latent variables to be ignored, reverting to a single-reward model. To overcome this limitation, we propose Swap-guided Preference Learning (SPL). The key idea is to construct fictitious swap annotators and use the mirroring property of their preferences to guide the encoder. SPL introduces three components: (1) swap-guided base regularization, (2) Preferential Inverse Autoregressive Flow (P-IAF), and (3) adaptive latent conditioning. Experiments show that SPL mitigates collapse, enriches user-specific latents, and improves preference prediction. Our code and data are available at https://github.com/cobang0111/SPL
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。