用少量标注数据+大量无标注数据，低成本实现语言模型对齐。
Semi-Supervised Preference Optimization with Limited Feedback
- 通过理论证明可设定奖励阈值，自动为无标注数据打伪标签。
- 仅用1%标注数据训练，效果超过基准模型用10%数据的结果。
- 适合资源有限但需保持人类偏好对齐的研究者使用。
偏好优化在提升语言模型与人类偏好对齐方面取得显著进展。然而，现有方法仍严重依赖大量成对标注反馈数据，导致资源消耗巨大。为此，本文研究半监督偏好优化（SSPO）问题，即同时利用少量成对偏好标签和大规模无成对样本进行学习。核心理论贡献是证明存在一个最优奖励阈值，能以高概率区分优劣响应，从而实现无标注数据的合理伪标签化。基于这些伪标签，SSPO可有效从海量无标注数据中提炼隐含偏好，维持人类对齐的同时大幅降低数据获取成本。跨多个数据集的实验验证了其卓越的数据效率：例如，使用仅1%的UltraFeedback数据，在Mistral-7B-Instruct上训练的SSPO始终优于在10%数据上训练的强大基线模型。
原文摘要 · Abstract (English)
The field of preference optimization has made outstanding contributions to the alignment of language models with human preferences. Despite these advancements, recent methods still rely heavily on substantial paired (labeled) feedback data, leading to substantial resource expenditures. To address these challenges, we study the problem of Semi-Supervised Preference Optimization (SSPO) in which the idea is to learn from both a small number of pairwise preference labels and a large pool of unpaired samples simultaneously. Our key theoretical contribution proves the existence of an optimal reward threshold capable of separating winning and losing responses with high probability, which enables a principled pseudo-labeling of unpaired data. By leveraging these pseudo-labels, SSPO effectively distills latent preferences from large-scale unpaired data, thus maintaining human alignment while drastically reducing acquisition costs. Extensive experiments across datasets validate this remarkable data efficiency; for instance, SSPO trained with Mistral-7B-Instruct on just 1% of UltraFeedback consistently surpasses strong baselines trained on 10% of UltraFeedback.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。