多大模型修订流程的增益来自重解、支架或内容,取决于任务类型。
Revision or Re-Solving? Decomposing Second-Pass Gains in Multi-LLM Pipelines
- 通过分解实验分离出重解、支架和内容三类增益成分。
- 在选择题中强模型直接回答比修订弱草稿更有效,代码任务中草稿结构有帮助。
- 草稿质量差时,弱评审模型反而更难提升,需针对性设计流水线。
多大模型修订流程普遍被认为通过修正错误带来提升,但我们通过受控分解实验,使用四组匹配条件将第二轮增益拆分为三类可加成分:重解、支架与内容。在两组模型对、三个涵盖知识密集型选择题与竞赛编程的任务上进行评估。结果表明,多模型修订的增益并非单一来源,而是依赖于任务结构、草稿质量及草稿信息类型。在选择题任务中,答案空间受限且草稿提供的结构引导有限,大部分增益源自更强模型的重解,直接将问题路由至强模型比修订弱草稿更高效。而在代码生成任务中,两阶段提示仍具价值,因为即使语义空白的草稿也能提供显著结构支架,而弱草稿内容可能有害。角色反转实验进一步显示,强草稿明显有利于弱评审模型。最终发现,多模型修订的效用动态受任务结构与草稿质量制约,需采用更精准的流水线设计,而非通用修订策略。
原文摘要 · Abstract (English)
Multi-LLM revision pipelines, in which a second model reviews and improves a draft produced by a first, are widely assumed to derive their gains from genuine error correction. We question this assumption with a controlled decomposition experiment that uses four matched conditions to separate second-pass gains into three additive components: re-solving, scaffold, and content. We evaluate this design across two model pairs on three benchmarks spanning knowledge-intensive MCQ and competitive programming. Our results show that the gains of multi-LLM revision are not monolithic, but depend on task structure, draft quality, and the type of draft information. On MCQ tasks, where the answer space is constrained and drafts provide little structural guidance, most gains are consistent with stronger-model re-solving, and directly routing queries to the stronger model can be more effective than revising a weak draft. On code generation tasks, however, two-stage prompting remains useful because even semantically null drafts can provide substantial structural scaffolding, while weak draft content can be harmful. Finally, role-reversed experiments show that strong drafts clearly benefit weak reviewers. Ultimately, our findings demonstrate that the utility of multi-LLM revision is dynamically bottlenecked by task structure and draft quality, necessitating more targeted pipeline designs rather than blanket revision strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。