用新指标提升大模型推理多样性,让同一问题有更多解法。
Reasoning Path Divergence: A New Metric and Curation Strategy to Unlock LLM Diverse Thinking
- 引入多解训练范式,打破单一标准答案限制。
- 提出路径发散度指标,量化多步推理过程差异。
- 实测推理结果更丰富,准确率提升超4.9%。
尽管测试时扩展(TTS)能有效提升大语言模型的推理能力,但模型输出缺乏多样性常成为瓶颈,这部分源于常见的“一题一解”(1P1S)训练方式,即仅提供单一标准答案,导致模型趋于固定推理路径。这种同质化不仅影响采样效率,也限制了后续强化学习阶段的探索空间。为此,本文提出“一题多解”(1PNS)训练范式,让模型接触多样有效的推理路径,从而提升推理多样性。核心挑战在于可靠度量多步思维链间的语义差异,因此提出一步级指标——推理路径发散度(RPD),通过对齐和评分长思维链来捕捉中间推理的差异。基于RPD,我们为每道题筛选最大多样性解集,并对Qwen3-4B-Base进行微调。实验表明,使用RPD筛选的训练数据可生成更丰富的输出,且在pass@k指标上表现更优:相比强基线平均提升+2.80%(pass@16),在AIME24上提升+4.99%,验证了1PNS能进一步增强TTS效果。
原文摘要 · Abstract (English)
While Test-Time Scaling (TTS) has proven effective in improving the reasoning ability of large language models (LLMs), low diversity in model outputs often becomes a bottleneck; this is partly caused by the common "one problem, one solution" (1P1S) training practice, which provides a single canonical answer and can push models toward a narrow set of reasoning paths. This homogenization not only limits sampling effectiveness but also restricts the exploration space for subsequent Reinforcement Learning (RL) stages. To address this, we propose a "one problem, multiple solutions" (1PNS) training paradigm that exposes the model to a variety of valid reasoning trajectories and thus increases inference diversity. A core challenge for 1PNS is reliably measuring semantic differences between multi-step chains of thought, so we introduce Reasoning Path Divergence (RPD), a step-level metric that aligns and scores Long Chain-of-Thought solutions to capture differences in intermediate reasoning. Using RPD, we curate maximally diverse solution sets per problem and fine-tune Qwen3-4B-Base. Experiments show that RPD-selected training yields more varied outputs and higher pass@k, with an average +2.80% gain in pass@16 over a strong 1P1S baseline and a +4.99% gain on AIME24, demonstrating that 1PNS further amplifies the effectiveness of TTS. Our code is available at https://github.com/fengjujf/Reasoning-Path-Divergence .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。