用树跳跃结构分析大模型推理过程,揭示其思维模式差异。
ReJump: A Tree-Jump Representation for Analyzing and Improving LLM Reasoning
- 将推理过程建模为树节点间的跳转行为,区分计算与回溯等动作。
- 发现准确率相近的模型推理路径差异显著,任务类型影响探索与利用平衡。
- 可用于改进推理质量,如指导最优生成和提示选择,适合模型可解释性研究者。
大型推理模型(LRMs)是专为生成长链思维(CoT)而训练的大语言模型,在数学与编程等挑战性任务上表现优异,但其内部推理机制仍不清晰。为此,我们提出ReJump,将推理轨迹表示为问题求解中间步骤树中节点的访问顺序。节点间转移称为‘跳转’,包括相邻移动(如计算)与非相邻移动(如回溯、验证)。ReJump支持多种度量指标,量化探索、利用、过度思考、遗忘与验证等行为。通过我们提出的LLM代理将推理轨迹转化为ReJump格式,我们在两个任务上评估了前沿LRMs,发现准确率相似的模型可能表现出截然不同的推理行为,且不同任务偏好不同的推理风格(如探索与利用的平衡)。进一步地,我们利用ReJump对比蒸馏后的LRMs与其教师模型、CoT提示的LLMs与LRMs,以及分析推理示例数量与强化学习对推理行为的影响。最后,我们展示通过ReJump引导的Best-of-N选择与提示选择策略,可在测试时提升推理质量。代码已公开于https://github.com/UW-Madison-Lee-Lab/ReJump。
原文摘要 · Abstract (English)
Large Reasoning Models (LRMs) are Large Language Models (LLMs) explicitly trained to generate long-form Chain-of-Thoughts (CoTs), achieving impressive success on challenging tasks like math and programming. However, their underlying reasoning "algorithms" remain poorly understood. To investigate this, we propose ReJump, which represents a reasoning trace as a visitation order over nodes in a tree of intermediate problem-solving steps. Transitions between nodes, which we term jumps, include adjacent moves that capture behaviors such as calculation, and non-adjacent moves that capture behaviors such as backtracking and verification. ReJump enables analyzing LLM reasoning with diverse metrics that quantify exploration, exploitation, overthinking, forgetting, and verification. Using our proposed LLM agent to extract reasoning traces into ReJump format, we evaluate state-of-the-art LRMs on two tasks and find that models with similar accuracy can exhibit distinct reasoning behaviors, while different tasks favor different reasoning styles (e.g., varying balance between exploration and exploitation). To further understand how learning strategies shape reasoning, we use ReJump to compare distilled LRMs with their teachers, CoT-prompted LLMs with LRMs, and to examine how the number of reasoning examples and reinforcement learning affect reasoning behavior. Finally, we show that ReJump can improve reasoning quality at test time through strategies such as ReJump-guided Best-of-N selection and prompt selection. Our code is publicly available at https://github.com/UW-Madison-Lee-Lab/ReJump.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。