让大模型个性化更可靠，通过概率化用户偏好捕捉不确定性。
Uncertainty-Aware Variational Reward Factorization via Probabilistic Preference Bases for LLM Personalization
- 用变分分布表示用户偏好，替代固定点估计。
- 在三个数据集上对新用户和低样本场景均有显著提升。
- 适合需要高可靠性个性化的推荐与对话系统开发者。
奖励因子分解通过将奖励分解为共享基函数和用户特定权重来实现大语言模型个性化。然而，现有方法在孤立地从稀缺数据中估计用户权重时，将其视为确定性点，导致推断不准确且不可靠。我们提出变分奖励因子分解（VRF），一种考虑不确定性的框架，将每个用户的偏好表示为共享偏好空间中的变分分布。VRF通过变分编码器推断用户分布，利用沃尔德斯坦距离匹配共享的概率基函数来推导权重，并通过方差衰减损失降低不确定估计的影响。在三个基准测试中，VRF在已见和未见用户、少样本场景及不同不确定性水平下均优于所有基线方法，其优势还延伸至下游对齐任务。代码开源于 https://github.com/Gyu-Seok-Lee/VRF_COLM26。
原文摘要 · Abstract (English)
Reward factorization personalizes large language models (LLMs) by decomposing rewards into shared basis functions and user-specific weights. Yet, existing methods estimate user weights from scarce data in isolation and as deterministic points, leading to inaccurate and unreliable inference. We introduce Variational Reward Factorization (VRF), an uncertainty-aware framework that represents each user's preferences as a variational distribution in a shared preference space. VRF infers user distributions via a variational encoder, derives weights through Wasserstein distance matching with shared probabilistic bases, and downweights uncertain estimates through a variance-attenuated loss. On three benchmarks, VRF outperforms all baselines across seen and unseen users, few-shot scenarios, and varying uncertainty levels, with gains extending to downstream alignment. Our code is available at https://github.com/Gyu-Seok-Lee/VRF_COLM26.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。