让大模型在保持对齐性的同时恢复多样回答能力。
Recovering Diversity Without Losing Alignment: A DPO Recipe for Post-Trained LLMs
- 通过重构基模型响应并构建偏好数据对，提升多样性。
- 在多个模型上使答案多样性提升达134%，同时保持性能不降。
- 适合需要高多样性且不牺牲安全性的生成场景。
许多开放指令存在多个有效回答，但后训练常使大模型输出趋于单一标准答案。本文提出 REDIPO，一种离线的 DPO 数据构建流程，可在保留指令对齐优势的前提下恢复多样有效回答模式。针对每个提示，REDIPO 从基础模型和指令模型中采样响应，用指令模型重写基模型输出，筛选安全且符合指令的质量候选，构建偏好对以偏好略微多样但指令遵循得分相近的回答。在 Qwen3-4B、OLMo-3-7B 和 LLaMA-3.1-8B 上，REDIPO 相较于指令检查点，NoveltyBench distinct_k 分别提升 134%、33% 和 44%；而 DivPO 在相同模型上分别变化 0%、-6%、-4%。这些改进基本维持 MTBench、IFEval 及 Arena-Hard 性能，并降低 HarmBench 攻击成功率。消融实验表明，边际多样性配对选择与基响应重写是多样性提升的关键，过滤与质量约束配对有助于保持对齐。结果表明，可通过精心构造的偏好数据重新引入基模型生成的有效多样化回答，同时保留后训练的对齐收益。代码与数据已开源。
原文摘要 · Abstract (English)
Many open-ended instructions have multiple valid answers that users can benefit from seeing, but post-training often narrows an LLM's output space toward a small set of canonical responses. We introduce REDIPO, an offline DPO data-construction pipeline for recovering distinct valid answer modes while preserving the alignment benefits of the instruct model. For each prompt, REDIPO samples responses from both base and instruct models, rewrites base-model responses with the instruct model, filters candidates for safety and instruction-following quality, and builds preference pairs that favor marginally diverse responses among candidates with similar instruction-following reward. Across Qwen3-4B, OLMo-3-7B, and LLaMA-3.1-8B, REDIPO improves NoveltyBench distinct_k by 134%, 33%, and 44% relative to the instruct checkpoints, while DivPO changes diversity by 0%, -6%, and -4% on the same models. These gains largely maintain MTBench, IFEval, and Arena-Hard performance, and reduce direct-category HarmBench attack success rate. Ablations show that marginal-diversity pair selection and base-response rewriting drive the diversity gains, while filtering and quality-bounded pairing help maintain alignment. Overall, our results show that diverse valid answers from base-model generations can be reintroduced through carefully constructed preference data while retaining the alignment benefits of post-training. We release our code and data at https://github.com/vsamuel2003/ReDiPO.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。