用模型自判方式实现低成本个性化对齐，无需额外训练。
Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgment
- 让模型自己判断生成内容是否符合偏好，避免外部奖励信号
- 实验显示该方法在保持高效的同时实现个性化对齐
- 适合需要快速适配用户偏好的场景，如对话系统
将语言模型与人类偏好对齐面临重大挑战，尤其是在不增加计算成本的前提下实现个性化。现有方法依赖奖励信号和额外标注数据，限制了其可扩展性和对多样人类价值观的适应性。为此，我们提出Persona-judge，一种新型判别范式，可在无需训练的情况下实现对未见偏好的个性化对齐。不同于通过外部奖励反馈优化策略参数，Persona-judge利用模型自身的偏好判断能力：一个草稿模型根据给定偏好生成候选词，而另一个体现不同偏好的裁判模型交叉验证这些词是否应被接受。实验表明，Persona-judge通过利用模型内在的偏好评估机制，提供了一种可扩展且计算高效的个性化对齐方案，为更灵活的定制化对齐开辟了道路。代码已公开。
原文摘要 · Abstract (English)
Aligning language models with human preferences presents significant challenges, particularly in achieving personalization without incurring excessive computational costs. Existing methods rely on reward signals and additional annotated data, limiting their scalability and adaptability to diverse human values. To address these challenges, we introduce Persona-judge, a novel discriminative paradigm that enables training-free personalized alignment with unseen preferences. Instead of optimizing policy parameters through external reward feedback, Persona-judge leverages the intrinsic preference judgment capabilities of the model. Specifically, a draft model generates candidate tokens conditioned on a given preference, while a judge model, embodying another preference, cross-validates the predicted tokens whether to be accepted. Experimental results demonstrate that Persona-judge, using the inherent preference evaluation mechanisms of the model, offers a scalable and computationally efficient solution to personalized alignment, paving the way for more adaptive customized alignment. Our code is available here.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。