让大模型分步优化推理，提升数学题解的逻辑性与准确性
HiPO: Hierarchical Preference Optimization for Adaptive Reasoning in LLMs
- 将回答拆分为提问澄清、推理步骤、答案三部分，逐段优化
- 7B模型在数学基准上表现优于DPO，GPT-4评估更逻辑清晰
- 兼顾训练稳定性和推理结构化，适合复杂任务微调
直接偏好优化（DPO）虽能对齐人类偏好，但在复杂推理任务中因缺乏细粒度反馈而表现不佳。现有方法或擅长稳定偏好学习（如KTO、RSO），或专注结构化推理（如ReMA、思维树），但难以融合。本文提出层次偏好优化（HiPO），将回答分解为查询澄清、推理步骤和答案三个模块，分别计算DPO损失并加权求和。该方法实现段级训练，保持DPO的高效与稳定。在Math Stack Exchange偏好数据集上微调多个7B模型后，HiPO模型在多种数学基准测试中优于DPO，且经GPT-4.1评估显示组织性、逻辑连贯性和一致性显著提升。
原文摘要 · Abstract (English)
Direct Preference Optimization (DPO) is an effective framework for aligning large language models with human preferences, but it struggles with complex reasoning tasks. DPO optimizes for the likelihood of generating preferred over dispreferred responses in their entirety and lacks the granularity to provide feedback on subsections of many-step solutions typical of reasoning tasks. Existing methods excel at either stable preference learning (e.g., DPO variants like KTO and RSO) or structured reasoning (e.g., ReMA's multi-agent RL framework, Tree of Thoughts), but fail to merge these complementary strengths. We propose HiPO (Hierarchical Preference Optimization), an extension of DPO that separates responses into reasoning segments (query clarification and context, reasoning steps, and answer) and computes loss as a weighted sum of the DPO loss for each segment. Our approach enables segment-specific training while maintaining DPO's computational efficiency and training stability. We demonstrate that for multiple 7B LLMs fine-tuned using HiPO and DPO on the Math Stack Exchange preference dataset, the models trained with HiPO outperform the others on a variety of common math benchmarks and achieve greater organization, logical flow, and consistency as measured by GPT-4.1.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。