arXiv:2608.00685cs.AI2026-08

对比多种LLM调度方法,发现增益有限且依赖模型,需权衡成本与效果。

When Does LLM Orchestration Pay Off? A Controlled Evaluation of Accuracy, Cost, and Task Difficulty

论文配图:When Does LLM Orchestration Pay Off? A Controlled Evaluation of Accuracy, Cost, and Task Difficulty
图 1 · 摘自论文原文
  • 在相同优化预算下比较五种模型的调度策略
  • 最高提升4.6个百分点,但耗时增加2到4倍
  • 不同模型表现差异大,需按模型定制调度方案

LLM调度常被认为能通过增加推理计算提升推理能力,但其收益可能无法抵消成本。现有对比常忽略优化投入差异,难以分离调度本身的价值。本文在五种LLM骨干模型和三个领域(编程竞赛、国际象棋谜题、数学)上,对Self-Refine、Best-of-$N$、Debate与仅任务和链式思维(CoT)单次调用基线进行受控评估。所有方法均在相同优化预算下使用GEPA优化,并在难度分层的基准数据上测试。调度带来适度但依赖基准的提升:在各基准内平均,最大提升为4.6个百分点(相比优化后的CoT)和4.5个百分点(相比仅任务基线),但总词元数约为仅任务推理的2至4倍。人类定义的任务难度与绝对准确率呈负相关,但在各基准内分析未显示调度效果随任务难度上升。探索性混合效应分析揭示三种基准中调度方法与模型间存在强交互作用,表明调度有效性高度依赖底层模型。结果建议调度决策应针对具体模型,并评估中等准确率提升是否值得额外推理成本。更广泛地,对LLM调度的评估应控制优化投入,并报告模型特定的准确率-成本权衡,而非将额外推理结构视为普遍有益。

原文摘要 · Abstract (English)

LLM orchestration is often assumed to improve reasoning by allocating additional inference-time computation, yet its gains may not justify its cost. Existing comparisons also frequently overlook differences in optimization effort, making it difficult to isolate the value of orchestration itself. We conduct a controlled evaluation of Self-Refine, Best-of-$N$, and Debate against task-only and chain-of-thought (CoT) single-call baselines across five LLM backbones and three domains: competitive programming, chess puzzles, and mathematics. For comparability, we optimize each method with GEPA under the same optimization budget and evaluate all methods on the same difficulty-stratified benchmark items. Orchestration yields moderate but benchmark-dependent gains: averaged across backbones within each benchmark, the largest improvement is 4.6 percentage points over optimized CoT inference and 4.5 points over task-only inference, while requiring approximately 2 to 4 times the mean total tokens of task-only inference. Human-derived difficulty is associated with lower absolute accuracy in all three benchmarks, but within-benchmark analyses do not indicate that orchestration effects increase with task difficulty. By contrast, exploratory mixed-effects analyses reveal strong interactions between orchestration method and backbone model across all three benchmarks, showing that orchestration effectiveness depends substantially on the underlying model. Our results suggest that orchestration decisions should be model-specific and account for whether moderate accuracy gains justify the additional inference cost. More broadly, evaluations of LLM orchestrations should control optimization effort and report model-specific accuracy--cost trade-offs rather than treating additional inference-time structure as uniformly beneficial.

LLM调度推理优化成本效益模型差异

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。