研究发现：辩论能救命，但难靠信号精准触发。
Statistical Scouting Finds Debate-Safe but Not Debate-Useful Cases: A Matched-Ceiling Study of Open-Weight LLM Reasoning Protocols
- 用投票熵判断是否该辩论，比固定策略更优
- 最优策略因模型和数据集而异，无通用解
- 现有简单信号无法有效识别真正需要辩论的案例
在每例输出限制为960个词元的条件下，研究比较了贪婪解码、三样本投票与双代理批判修正辩论在MuSiQue和GSM8K数据集上的表现。结果显示，若能为每例选择最佳协议（即理想路由），性能可比最优固定策略分别提升14.0和13.7个百分点。但不同模型与数据集组合中，最佳协议各不相同。仅投票熵阈值在两个模型上方向性优于最优固定策略（+1.3和+1.7个百分点），但统计显著性不足。联合分析显示方向一致但未达显著水平。学习型控制器（逻辑回归、梯度提升树）未超越阈值。核心发现是：高投票熵可避免辩论反效果，但66%真正需要辩论的情况发生在投票一致却错误时。对Llama进行单次提示自检，在全部127个一致案例中均改变答案，表明其与需辩论标签无互信息，可能受提示服从性干扰，无法作为有效路由信号。恢复剩余性能差距需规避格式服从混淆的行为探测器。
原文摘要 · Abstract (English)
When should a language model answer directly, sample and vote, or engage in multi-agent debate? Recent work shows voting often explains much of the gain attributed to debate, while selective-debate systems activate deliberation only on uncertain examples. We ask: under a matched ceiling on generated tokens (960 per example), how much per-example routing headroom exists, and how much is recoverable from cheap pre-deliberation signals? We evaluate greedy decoding, three-sample voting, and a two-agent critique-revise debate on MuSiQue and GSM8K using Llama 3.1 8B Instruct and Ministral 3 8B Instruct. On MuSiQue, an oracle selecting the correct protocol per example gains +14.0 and +13.7 pp over the best fixed one. The best fixed protocol is model- and dataset-dependent: each (model, dataset) cell has a different winner. This headroom is hard to recover from cheap ex-ante signals. A vote-entropy threshold is the only controller that directionally beats the best fixed protocol on both models (+1.3 and +1.7 pp), though individual paired-bootstrap CIs include zero. A joint analysis (meta-analysis +1.6 pp, p=0.125; Bayesian P(both>0)=0.59) is directionally consistent but not significant. Learned controllers (LR, GBT) do not outperform the threshold. The key finding is structural: vote entropy predicts where debate is safe, not where debate is needed. High entropy sharply reduces debate backfire, but 66% of debate-helpful examples (31/47) occur when voting is unanimous but wrong. A single-prompt self-critique probe on Llama flips the answer in 127/127 unanimous cases, yielding zero mutual information with the debate-helpful label; we cannot rule out a prompt-compliance artifact, but either interpretation disqualifies the probe as a router. Recovering the remaining headroom requires behavioral probes that avoid format-compliance confounds at the 8B scale.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。