让大模型更对齐人类意图，需用数值反馈代替二选一偏好
Beyond Ordinal Preferences: Why Alignment Needs Cardinal Human Feedback
- 用金钱意愿法收集用户对响应质量的数值评价
- 引入数值反馈后模型在难题集上表现超越传统方法
- 证明仅靠二选一选择无法真正找到最优模型
大语言模型对齐技术依赖偏好优化，通常通过二元选择获取用户的序数偏好。然而我们发现更根本的问题：这类方法收集的是错误类型的数据。理论上证明，仅依赖序数比较的算法无法系统性地识别出最符合偏好的模型。直观上，序数数据缺乏解决权衡的信息——例如，修复一个提示的事实错误，与提升另一个提示的风格之间如何取舍。要选出最优模型，必须恢复对模型本身的偏好，而这只能通过关于响应质量的基数反馈来实现。为此，我们采用实验经济学中的愿意支付法，收集并公开发布了一个包含25,000条基数判断的数据集。实证结果表明，将基数反馈融入偏好微调，可使模型更聚焦于高影响力改进，在下游基准测试（如Arena-Hard）中表现优于仅使用序数反馈的方法。
原文摘要 · Abstract (English)
Alignment techniques for LLMs rely on optimizing preference-based objectives -- where these preferences are typically elicited as ordinal, binary choices between responses. Recent work has focused on improving label quality or mitigating particular biases, but we identify a more fundamental limitation: these methods collect the wrong kind of data. We prove an impossibility result: no algorithm relying solely on ordinal comparisons can systematically recover the most preferred model. Intuitively, ordinal data lacks the information needed to resolve tradeoffs -- e.g., fixing a factual error on one prompt versus improving style on another. We show that selecting the optimal model requires recovering preferences over \emph{models} (rather than just responses), which can only be identified given cardinal feedback about response quality. To address this, we collect and publicly release a dataset of 25,000 cardinal judgments using willingness-to-pay elicitations, a well-established tool from experimental economics. Empirically, we find that incorporating cardinal feedback into preference fine-tuning allows models to prioritize high-impact improvements and outperform ordinal-only methods on downstream benchmarks, such as Arena-Hard.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。