用轻量模型调度多个大模型解决前沿科学推理难题,效率更高、成本更低。
SciOrch: Learning to Orchestrate Expert LLMs for Solving Frontier Multimodal Scientific Reasoning Tasks

- 训练一个80亿参数模型,通过分解问题并调用API分配任务给不同大模型
- 在240题测试集上准确率达56.66%,比最强单模型高3.74%,成本不足多模型方法一半
- 适合需要高精度科学推理且关注算力与费用的科研人员和工程团队
前沿科学推理仍是大型语言模型的重大挑战,即使最强的商用系统也难以达到专家水平。分析模型行为发现,不同前沿模型在不同题型上表现互补,单一模型无法覆盖全部能力。我们提出SciOrch框架,训练一个80亿参数的轻量级模型来协调多个前沿大模型进行科学推理。该协作者将问题分解,通过API调用将子问题分派给选定的商用模型,并整合结果生成最终答案。训练此类协作者极具挑战:每次操作触发昂贵的API调用,导致传统在线强化学习不可行。我们采用基于蒙特卡洛树搜索的方法,生成多样化调度路径,提取节点级单轮样本,使用类似GRPO的训练策略优化协作者。在包含SGI-Reasoning和Scientists' First Exam的240题测试集上,SciOrch平均准确率达56.66%,比最强单模型高出3.74%,比最强多代理基线高3.33%。同时在SGI和SFE上均取得最佳成绩,且API开销不足典型多代理方法的一半。
原文摘要 · Abstract (English)
Frontier scientific reasoning remains a major challenge for large language models (LLMs), where even the strongest commercial systems fall short of expert-level performance. A closer look at model behavior reveals substantial complementarity that single-model evaluation hides: different frontier models excel on different question types, and no single model captures the full picture. We present SciOrch, a framework that trains a lightweight 8B model to orchestrate frontier LLMs for scientific reasoning. The orchestrator decomposes each question, delegates sub-problems to selected commercial models through API calls, and synthesizes a final answer. Training such an orchestrator is fundamentally harder than conventional agentic RL: each action triggers an API call that is expensive in both dollar cost and latency, making standard online rollouts infeasible. We address this with MCTS-based approach, producing diverse orchestration trajectories, extracting per-node single-turn samples, and optimizing the orchestrator with GRPO-style training. On a 240-question test set spanning SGI-Reasoning and Scientists' First Exam, SciOrch reaches 56.66% average accuracy, outperforming the strongest single commercial model by 3.74% and the strongest multi-agent baseline by 3.33%. It also attains the best accuracy on both SGI and SFE with less than half the API cost of typical multi-agent methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。