通过对比反馈高效学习多维用户偏好，提升大模型个性化生成能力。
Comparison-based Active Preference Learning for Multi-dimensional Personalization
- 基于贝叶斯推理与主动查询策略，从用户对比反馈中推断隐式偏好。
- 实验表明，仅需少量比较即可实现高质量个性化生成，显著降低用户负担。
- 适合需要高效率个性化交互的应用场景，如对话系统与内容推荐。
大型语言模型（LLMs）虽取得显著进展，但与其对齐人类偏好仍是核心挑战。由于个体偏好具有多维度特征，近期研究探索了多维个性化，旨在使模型生成符合显式偏好的回应。然而，人类偏好常为隐式且难以言明，限制了该方法的直接应用。为此，我们提出主动多维偏好学习框架AMPLe，通过交互式收集对比反馈来捕捉隐式用户偏好。基于贝叶斯推断，我们改进后验更新机制以缓解比较中的估计偏差与噪声问题；同时受广义二分搜索启发，采用主动查询策略最小化所需比较次数。理论分析与语言生成任务实验均验证了该框架在反馈效率与个性化效果上的优越性。代码已开源：https://github.com/ml-postech/AMPLe。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown remarkable success, but aligning them with human preferences remains a core challenge. As individuals have their own, multi-dimensional preferences, recent studies have explored multi-dimensional personalization, which aims to enable models to generate responses personalized to explicit preferences. However, human preferences are often implicit and thus difficult to articulate, limiting the direct application of this approach. To bridge this gap, we propose Active Multi-dimensional Preference Learning (AMPLe), designed to capture implicit user preferences from interactively collected comparative feedback. Building on Bayesian inference, our work introduces a modified posterior update procedure to mitigate estimation bias and potential noise in comparisons. Also, inspired by generalized binary search, we employ an active query selection strategy to minimize the number of required comparisons by a user. Through theoretical analysis and experiments on language generation tasks, we demonstrate feedback efficiency and effectiveness of our framework in personalizing model responses. Our code is publicly available at https://github.com/ml-postech/AMPLe .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。