测试大模型能否协作推理,发现现成模型难有效合作。
Off-Trajectory Reasoning: Can LLMs Collaborate on Reasoning Trajectory?
- 设计两种测试:回溯干扰和接受引导,评估模型协作能力。
- 所有模型在复杂任务中协作成功率低于9.2%,强者反而更脆弱。
- 训练方式影响协作效果,教师模型缺陷会传递给学生模型。
推理型大模型通过显式表达思考过程,在复杂任务上表现优异。这种透明性为多模型在共享推理路径中直接协作提供了可能,有望提升推理效率与探索能力。关键前提是能评估并利用他人部分思考——我们称之为离轨推理。本文探究标准单模型训练是否具备此类能力。提出两种测试:恢复性(Recoverability)测试模型从误导性推理中回溯的能力,可引导性(Guidability)测试其基于强模型正确引导进行推进的能力。评估15个开源大模型(1.5B-32B),结果出人意料:基准表现更强的模型在干扰下反而更脆弱;所有模型在超出自身能力的问题上,无法有效利用协作引导,解决率均低于9.2%。进一步控制实验分离了后训练中的三个因素:教师模型选择、强化学习使用、数据选择策略的影响。结果显示,即使蒸馏轨迹正确,教师模型的低恢复性行为仍会传递给学生模型。本研究为多模型协同推理提供评估框架,揭示了现有推理模型在协作上的根本局限。
原文摘要 · Abstract (English)
Reasoning LLMs are trained to verbalize their reasoning process, yielding strong gains on complex tasks. This transparency also opens a promising direction: multiple reasoners can directly collaborate on each other's thinking within a shared trajectory, yielding better inference efficiency and exploration. A key prerequisite, however, is the ability to assess the usefulness and build on another model's partial thinking -- we call this off-trajectory reasoning. Our paper investigates a critical question: can standard solo-reasoning training pipelines deliver desired off-trajectory behaviors? We propose twin tests that capture the two extremes of the off-trajectory spectrum, namely Recoverability, which tests whether LLMs can backtrack from "distractions" induced by misleading reasoning traces, and Guidability, which tests their ability to build upon correct reasoning from stronger collaborators. Our study evaluates 15 open-weight LLMs (1.5B-32B) and reveals a counterintuitive finding -- "stronger" LLMs on benchmarks are often more fragile under distraction. Moreover, all models tested fail to effectively leverage guiding steps from collaborators on problems beyond their inherent capabilities with solve rates remaining under 9.2%. Finally, we conduct control studies to isolate the effects of three factors in post-training on these behaviors: the choice of distillation teacher, the use of RL, and data selection strategy. Our results provide actionable insights for training natively strong reasoning collaborators; e.g., we find that suboptimal recoverability behaviors of teacher models are transferred to distilled students even if the distillation trajectories are correct. Taken together, this work lays the groundwork for evaluating multi-model collaborations in shared reasoning trajectories and highlights the limitations of off-the-shelf reasoning LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。