用统计检验动态控制大模型辩论轮次,省计算又防失败。
Sequential Consensus for Multi-Agent LLM Debates: A Wald-SPRT compute governor with calibration-based failure detection

- 引入沃尔德检验,根据判断得分动态决定辩论是否停止
- GSM8K上平均1.01轮就停,减少3.7倍调用,仅损失2个百分点准确率
- 适合需要高效部署的大模型多智能体系统,尤其关注成本与可靠性
多智能体大模型辩论能提升事实性和推理能力,但多数方法固定辩论轮数,导致简单任务浪费算力、难题又不够深入。本文将沃尔德的序贯概率比检验(SPRT)作为可插拔的算力控制机制:每轮后由大模型裁判给出[0,1]共识分,沃尔德监控器在贝塔分布假设下累积对数似然比,当达到边界或超过最大轮数R_max时停止,返回最优结果。在独立同分布假设下,该规则保持第一类/第二类错误率保证;实际部署中,校准结果更为关键,它衡量裁判得分能否有效区分有用收敛与无效进展。我们评估了两个方向:(i) 基于校准贝塔模型的蒙特卡洛研究,分析工作曲线、误差率、截断行为和敏感性;(ii) 在200个MMLU和200个GSM8K题目上进行真实大模型测试,使用三个异构代理(gpt-5, claude-opus-4-6, gemini-2.5-pro)和一个claude-opus-4-6裁判,使用互不重叠的40题校准子集。在GSM8K上,平均1.01轮停止(4.06次调用),准确率达97.0%,对比固定5轮的99.0%准确率(15次调用),实现3.7倍调用减少,仅-2个百分点精度损失;在MMLU上,校准后的KL趋近于0,规则在99.5%的题目上于2.1倍成本内截断。核心启示并非更准,而是经典序贯检验可作为低成本算力调控与失败检测层,适用于多智能体大模型系统。
原文摘要 · Abstract (English)
Multi-agent LLM debate improves factuality and reasoning, but most recipes pick a fixed round count, over-spending on easy items and under-spending on hard ones. We adapt Wald's Sequential Probability Ratio Test (SPRT) as a plug-in compute governor for LLM debates. After each round, an LLM judge emits a [0,1] consensus score on the latest agent positions; a Wald monitor accumulates the log-likelihood ratio of "useful convergence" vs "not yet useful" under a Beta likelihood family, and stops when either boundary is crossed or returns a capped best-effort outcome at R_max. Under i.i.d. assumptions the rule inherits SPRT type-I/type-II error guarantees; in deployment the calibration itself is the more important object, since it estimates whether the judge score actually separates useful from unhelpful convergence in a given domain. We evaluate two tracks: (i) a Monte-Carlo study under calibrated Beta models characterising working curves, error rates, capping behaviour, and sensitivity; and (ii) a real-LLM evaluation on 200 attempted MMLU and 200 attempted GSM8K items with three heterogeneous agents (gpt-5, claude-opus-4-6, gemini-2.5-pro) and a claude-opus-4-6 judge, using disjoint 40-item calibration subsets. On GSM8K the rule stops in 1.01 average rounds (4.06 LLM calls) at 97.0% accuracy vs 99.0% for fixed-5 debate at 15 calls: a 3.7x call reduction at -2pp accuracy. On MMLU the calibrated KL collapses to about 0 and the rule caps on 99.5% of items at 2.1x cost. The takeaway is not that SPRT makes debate more accurate, but that a classical sequential test serves as a cheap compute-control and failure-detection layer for multi-agent LLM systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。