用参考答案相似度替代人类偏好数据，实现高效模型对齐
Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Data
- 以生成结果与参考答案的相似度作为奖励信号
- 无需二元偏好数据和奖励模型，性能接近现有方法
- 适用于安全、置信度等多场景对齐，灵活易扩展
大型语言模型需具备有益性、无害性和诚实性。当前对齐方法依赖资源密集的二元偏好数据与奖励建模，而本工作探索利用采样生成结果与参考答案之间的相似度作为补充奖励函数。当存在单个参考答案时，该相似度奖励可避免使用二元偏好数据和显式奖励建模。我们提出 RefAlign——一种基于 REINFORCE 框架的通用对齐算法，不依赖奖励模型或参考模型。RefAlign 使用 BERTScore 等语言生成评估指标，计算生成结果与参考答案的相似度作为代理奖励。除通用偏好优化外，通过结合任务特定目标，可自然拓展至安全性和置信度对齐等场景。在多个任务中，RefAlign 在无二元偏好数据和奖励模型的情况下，性能与先前方法相当。代码已公开于 https://github.com/mzhaoshuai/RefAlign。
原文摘要 · Abstract (English)
Large language models~(LLMs) are expected to be helpful, harmless, and honest. In different alignment scenarios, such as safety, confidence, and general preference alignment, binary preference data collection and reward modeling are resource-intensive but play a central role in transferring human preferences. In this work, we explore using the similarity between sampled generations and reference answers as a supplementary reward function for alignment. When unary reference answers are available, such similarity-based rewards can circumvent the need for binary preference data and explicit reward modeling. We introduce \textit{RefAlign}, a versatile REINFORCE-style alignment algorithm that does not rely on reward or reference models. RefAlign utilizes language generation evaluation metrics, such as BERTScore, between sampled generations and reference answers as surrogate rewards. Beyond general preference optimization, RefAlign can be naturally extended to diverse scenarios, including safety and confidence alignment, by combining similarity-based rewards with task-specific objectives. Across multiple scenarios, RefAlign achieves performance comparable to prior alignment methods while operating without binary preference data or reward models. The code is available at https://github.com/mzhaoshuai/RefAlign.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。