设计文本评分规则，让AI生成内容更符合人类偏好。
Aligned Textual Scoring Rules
- 通过最小化与人工评分的均方误差优化文本评分规则。
- 在保持概率正确性的同时，提升与人类评分的一致性。
- 适合需要对齐人类偏好的文本生成评估场景。
评分规则通过将预测值与真实状态对比来获取概率性预测。若从代理视角看，报告真实信念能最大化期望得分，则该规则为正则。随着语言模型的发展，Wu 和 Hartline（2024）提出将文本信息获取问题转化为数值（即概率）信息获取问题，实现了文本获取的可证明正则性。然而，并非所有正则评分规则都与人类对文本的偏好一致。本文设计了对齐评分规则（ASR），通过优化和最小化正则评分规则与参考评分（如人工评分）之间的均方误差，使评分更贴合人类偏好。实验表明，该方法在保持正则性的同时，显著优于现有方法，在对齐人类偏好方面表现更优。
原文摘要 · Abstract (English)
Scoring rules elicit probabilistic predictions from a strategic agent by scoring the prediction against a ground truth state. A scoring rule is proper if, from the agent's perspective, reporting the true belief maximizes the expected score. With the development of language models, Wu and Hartline (2024) proposes a reduction from textual information elicitation to the numerical (i.e. probabilistic) information elicitation problem, which achieves provable properness for textual elicitation. However, not all proper scoring rules are well aligned with human preference over text. Our paper designs the Aligned Scoring rule (ASR) for text by optimizing and minimizing the mean squared error between a proper scoring rule and a reference score (e.g. human score). Our experiments show that our ASR outperforms previous methods in aligning with human preference while maintaining properness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。