让大模型像人一样多想几种解法，再选最可靠的。
Hint-Guided Diversified Policy Optimization for LLM Reasoning
- 先列多种解题思路（提示），再选出最可靠的一种
- 在数学推理任务上提升准确率，解法多样性显著增加
- 适合需要多角度思考的复杂推理场景
大语言模型在推理任务中表现日益出色，强化学习结合可验证奖励（RLVR）是有效提升策略。但现有方法仅基于结果正确性给予奖励，缺乏引导模型探索多样化解法的信号。人类解题常会考虑多种路径再择优选择，而当前框架未显式激励此过程。为此，我们提出提示引导的多样化策略优化（HDPO），使模型先生成多个候选解题思路作为提示，再从中筛选最可靠的进行深入推理。HDPO包含两个阶段：结构化推理冷启动与提示引导的多样化强化学习，推动模型遵循“提出-选择-思考”路径。实验表明，该方法显著提升大模型推理能力，增强解法多样性，并提高可靠解的识别能力。
原文摘要 · Abstract (English)
Recent developments in Large Language Models (LLMs) have showcased impressive reasoning capabilities, with Reinforcement Learning with Verifiable Rewards (RLVR) being a promising enhancement strategy. However, existing reward mechanisms are constrained to the outcome-level correctness and lack explicit signals to guide the model to consider diverse solutions. In contrast, human problem solving typically involves evaluating multiple potential approaches and selecting the most reliable solution, a cognitive process that current RLVR frameworks do not explicitly incentivize. Inspired by this, we propose Hint-Guided Diversified Policy Optimization (HDPO), allowing the model to first list all potential candidate solution outlines as hints and then select the most reliable one for further reasoning. HDPO comprises two stages of Cold Start for Structured Reasoning and Hint-Guided Diversified Reinforcement Learning to incentivize the model to generate diverse and reliable solutions following the ``propose-select-think'' trajectory. Experimental results show that HDPO effectively boosts LLM reasoning and enhances the diversity of candidate solutions as well as the LLM's ability to identify reliable solutions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。