优化指令质量可显著提升大模型对齐效果
Instruction Quality Matters: Refining Instructions for Effective Preference Learning
- 用奖励信号筛选弱指令，再通过规则引导的LLM反馈改进
- 在多个模型和基准上，改进后数据使对齐性能全面提升
- 适合关注指令工程与偏好学习的数据质量研究者
偏好学习通过成对响应优化模型，但这些响应的信息量本质上由生成它们的指令决定。本文识别出指令质量是偏好学习中的隐性瓶颈：低质量或模糊的指令会限制响应质量分布，导致优质选择响应不足，削弱偏好信号。通过Best- and Worst-of-N分析，我们发现指令质量同时影响采样响应质量的上限和下限。基于此，提出一种指令精炼流程：利用奖励信号识别弱指令，并通过规则引导的LLM反馈进行修订，从而在不丢弃样本的前提下提升偏好数据质量。在多个模型和基准上的离线与在线偏好学习实验表明，该方法在性能上优于原始数据和其它数据改进策略。进一步分析显示，指令精炼提升了可达到的响应质量，并能与以响应为中心的数据清洗相辅相成。总体而言，指令质量是决定大模型对齐中偏好信号有效性的重要因素。代码已开源：https://github.com/01choco/instruction-refinement/
原文摘要 · Abstract (English)
Preference learning optimizes models using response pairs, yet the informativeness of these pairs is fundamentally shaped by the instructions from which they are generated. We identify instruction quality as a hidden bottleneck in preference learning: low-quality or ambiguous instructions restrict the response-quality distribution, limiting strong chosen responses and weakening preference signals. Through Best- and Worst-of-N analyses, we show that instruction quality constrains both the ceiling and floor of sampled response quality. Motivated by this observation, we introduce an instruction-refinement pipeline that selects weak instructions using reward signals and revises them with rubric-guided LLM feedback, improving preference data without discarding examples. Across offline and online preference learning settings, experiments on multiple models and benchmarks show broad alignment improvements over original data and alternative data-improvement strategies. Further analyses indicate that instruction refinement raises achievable response quality and complements response-centric preference data curation. Overall, instruction quality emerges as a key factor governing how informative preference signals are formed for LLM alignment. Code is available at: https://github.com/01choco/instruction-refinement/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。