用几十条数据实现模型生成个性化，无需训练
Drift: Decoding-time Personalized Alignments with Implicit User Preferences
- 解码时通过少量示例建模用户偏好属性，动态调整输出
- 仅需50-100条样本，效果超过传统强化学习方法
- 无需训练、可解释性强，适合快速适配真实用户
个性化语言模型生成一直是大模型的重要目标。我们提出Drift框架，在解码阶段基于隐式用户偏好实现模型个性化。传统基于人类反馈的强化学习（RLHF）需数千条标注数据和昂贵梯度更新，而Drift采用无训练方式，仅需数十条示例即可通过高效偏好建模引导冻结模型。该方法将用户偏好建模为预定义的可解释属性组合，并在解码时进行对齐，实现个性化生成。在合成人物数据集(Perspective)和真实人工标注数据集(PRISM)上的实验表明，Drift显著优于RLHF基线，且仅使用50-100条样本。结果与分析显示，Drift兼具计算效率与可解释性。
原文摘要 · Abstract (English)
Personalized alignments for individual users have been a long-standing goal in large language models (LLMs). We introduce Drift, a novel framework that personalizes LLMs at decoding time with implicit user preferences. Traditional Reinforcement Learning from Human Feedback (RLHF) requires thousands of annotated examples and expensive gradient updates. In contrast, Drift personalizes LLMs in a training-free manner, using only a few dozen examples to steer a frozen model through efficient preference modeling. Our approach models user preferences as a composition of predefined, interpretable attributes and aligns them at decoding time to enable personalized generation. Experiments on both a synthetic persona dataset (Perspective) and a real human-annotated dataset (PRISM) demonstrate that Drift significantly outperforms RLHF baselines while using only 50-100 examples. Our results and analysis show that Drift is both computationally efficient and interpretable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。