大模型能预判失败风险，但难选哪种协作方式最划算。
LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off: Cost-Aware Protocol Routing Across Reasoning Tasks
- 用四种协作模式测试数学难题，对比不同决策策略的性价比
- 模型自信心可有效触发协作，但无法精准判断哪种协作更优
- 现有方法对特定协作方案的预测能力很弱，仍是未解难题
多智能体大语言模型系统可通过增加计算提升推理效果，但部署时需判断额外协作是否值得。本文固定求解器，在四种协议下测试同一问题：直接求解（基准）、迭代自修正（单机）、规划-执行-评审协作（PER）和多智能体讨论（广播）。主基准包含4,181道竞赛级数学题；稳健性验证覆盖四个涵盖数学、生物与广义科学的基准，使用两种求解器家族。在固定策略、训练路由和冻结LLM路由下，保守策略低估协作需求，而高求解率的冻结路由常过度启用协作。在回答后、协作前使用gpt-oss-120b探针，对基准失败的预测达到0.8847 AUROC（4,151个可解析案例；95%置信区间[0.8732, 0.8955]）。该指标对任何协作是否有效有信息量（0.7683 AUPRC），但对PER或广播具体价值的预测能力极弱（分别为0.1674和0.1041 AUPRC）。另，回答前的自信心门控可实现78.0%求解率（45K tokens），优于冻结gpt-oss-120b路由的73.8%（71.3K tokens），但低于事后固定顺序的最优门控（92.4%）。在10组配对设置中，最优门控比基准提升23.2–58.3分，但协议表现因任务而异。在六组预留路由评估中，最优门控仍领先18.5–28.9分。结果表明，自信心可支持初始协作触发，但针对具体协议的成本感知路由仍无解。
原文摘要 · Abstract (English)
Multi-agent large language model (LLM) systems can improve reasoning by spending more computation, but deployment requires deciding when extra collaboration is worth its cost. We isolate this decision by running every problem under four protocols while holding the solver fixed within each setting: direct solving (Baseline), iterative self-correction (Single), planner-executor-reviewer collaboration (PER), and multi-agent deliberation (Broadcast). The primary benchmark comprises 4,181 competition-level math problems; paired robustness checks cover four benchmarks spanning competition math, biology, and broader science with two solver families. Across fixed policies, trained routers, and frozen LLM routers, conservative policies under-escalate, whereas higher-solve frozen routers often over-escalate. A post-answer, pre-collaboration gpt-oss-120b probe ranks Baseline failures with 0.8847 AUROC (4,151 parseable cases; 95% CI [0.8732, 0.8955]). The same score remains informative for predicting whether any collaboration helps (0.7683 AUPRC), but is much weaker for identifying PER- or Broadcast-specific value (0.1674 and 0.1041 AUPRC). Separately, the pre-answer self-confidence gate reaches 78.0% solve at 45K tokens, compared with 73.8% at 71.3K for a frozen gpt-oss-120b router and 92.4% for a retrospective fixed-order oracle. Across 10 paired model-condition settings, the oracle adds 23.2-58.3 points of retrospective coverage over Baseline, but protocol profiles vary by task. In the six settings with held-out router evaluations, oracle gaps remain 18.5-28.9 points. Confidence can therefore support initial escalation, while protocol-specific cost-aware routing remains unresolved.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。