用语义标注动态调整每对样本的偏好优化温度，提升模型训练效果。
SP^2DPO: An LLM-assisted Semantic Per-Pair DPO Generalization
- 基于语义差距标注为每对数据分配专属温度参数
- 在AlpacaEval 2.0上实现更优的长度控制胜率，且无需调参
- 支持大规模可审计训练，无额外计算开销
直接偏好优化（DPO）通过单一全局温度参数平衡拟合偏好标签与保持与参考模型接近，但隐含假设所有偏好对信息量相同。真实偏好数据集包含高信号（如安全、事实性）与低信号（如风格）混杂，且存在标注噪声。本文提出SP²DPO（语义感知的逐对DPO），将全局温度替换为由教师模型生成的结构化语义差距标注（类别、幅度、置信度）预定义的实例级温度β_i。我们在UltraFeedback数据集（59,960对）上实现大规模β_i构建，训练时零开销：内循环优化器仍为标准DPO，仅按对设置β。实验聚焦AlpacaEval 2.0，报告原始胜率与长度控制胜率。在四个4B–8B开源指令微调学生模型上，SP²DPO性能优于调优后的全局β基线，在其中两个模型上提升长度控制胜率，且无需针对每模型调参。代码、标注与成果均公开。
原文摘要 · Abstract (English)
Direct Preference Optimization (DPO) controls the trade-off between fitting preference labels and staying close to a reference model using a single global temperature beta, implicitly treating all preference pairs as equally informative. Real-world preference corpora are heterogeneous: they mix high-signal, objective failures (for example, safety, factuality, instruction violations) with low-signal or subjective distinctions (for example, style), and also include label noise. We introduce our method, SP2DPO (Semantic Per-Pair DPO), a generalization that replaces the global temperature with an instance-specific schedule beta_i pre-decided offline from structured semantic-gap annotations (category, magnitude, confidence) produced by teacher language models. We instantiate this procedure on the UltraFeedback preference corpus (59,960 pairs), enabling large-scale construction of an auditable beta_i artifact, and incur zero training-time overhead: the inner-loop optimizer remains standard DPO with beta set per pair. We focus our empirical study on AlpacaEval 2.0, reporting both raw win rate and length-controlled win rate. Across four open-weight, instruction-tuned student backbones (4B-8B), SP2DPO is competitive with a tuned global-beta DPO baseline and improves AlpacaEval 2.0 length-controlled win rate on two of four backbones, while avoiding per-model beta sweeps. All code, annotations, and artifacts will be released.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。