提出可量化评估大模型协作效果的诊断框架，解决何时该用协作的问题。
Oracle Gap and Signal Fidelity: A Fixed-Pool Diagnostic for Test-Time Collaboration
- 将协作效果分解为可测量的四个因素：纠错潜力、信号覆盖、选择质量与误伤率。
- 实测显示协作增益受限于模型与任务的差距（oracle gap）和判断信号清晰度。
- 适合做模型部署前评估，尤其对想避免无效协作的研究者或工程师有用。
测试时协作（如自洽性、最佳N选一、评议员模型、验证流水线）常被认为能提升大模型推理能力，但其增益不均，有时甚至为负。本文针对固定候选池场景，将选择器或验证器的净收益分解为可测量因素：可恢复质量、验证信号覆盖率、条件选择质量及对正确输出的损害。这将协作重新理解为候选选择问题，而非多智能体拓扑的内在属性。在LiveCodeBench、MATH Level-5高难度题型和GPQA-Diamond上，性能上限首先由模型与最优解之间的差距（oracle gap）决定，其次取决于信号保真度——即验证结果与标准答案的候选级别一致性。在LiveCodeBench中，公开测试验证器（MCC 0.825）相比首个样本基线提升+8.14个百分点；生成测试验证器（MCC 0.248）提升+2.70个百分点，与大模型选择器无显著差异，但损害率接近零，而后者达4.69%。在MATH上，基于符号等价的选择器比自洽性高出+4.67个百分点，而大模型选择器反而为负。在GPQA-Diamond上，可恢复质量仅3.03%，87.54%的候选池答案相同；弱模型的候选池进一步缩小，表明oracle gap是任务、模型与采样配置的联合属性。本框架提供实用的部署前诊断工具：先估算oracle gap，再测量覆盖率、信号保真度与损害率，再决定是否投入协作。
原文摘要 · Abstract (English)
Test-time collaboration, including self-consistency, best-of-N selection, critic models, and verifier pipelines, is often credited with broadly improving LLM reasoning, yet its gains are uneven and sometimes negative. We ask when training-free collaboration should be expected to help. For a fixed candidate pool, we decompose a selector or verifier's net gain into measurable factors: recoverable mass, verification-signal coverage, conditional selection quality, and harm to already-correct outputs. This reframes collaboration as a candidate-selection problem rather than as an intrinsic property of a multi-agent topology. Across LiveCodeBench, MATH Level-5 hard subjects, and GPQA-Diamond, gains are bounded first by the oracle gap and then by signal fidelity, which we measure directly as candidate-level agreement between verifier verdicts and official labels. On LiveCodeBench, a public-test verifier (MCC 0.825) gains +8.14 percentage points (pp) over a first-sample baseline; a generated-test verifier (MCC 0.248) improves by +2.70pp and is not statistically distinguishable from an LLM selector, but operates at near-zero harm versus the selector's 4.69% harm rate. On MATH, a symbolic answer-equivalence selector beats self-consistency by +4.67pp, while LLM selectors are negative. On GPQA-Diamond, recoverable mass is only 3.03% and 87.54% of candidate pools are answer-identical; a weaker model's pools shrink both further, suggesting that oracle gap is a joint property of task, model, and sampling configuration. Our framework yields a practical pre-deployment diagnostic: estimate the oracle gap, then measure coverage, signal fidelity, and harm before investing in collaboration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。