让多个AI评委辩论，自动判断何时结束，准确率远超简单投票。
Multi-Agent Debate for LLM Judges with Adaptive Stability Detection
- 多个AI评委通过持续辩论修正判断，提升决策质量。
- 实验显示准确率高于多数投票法，且节省计算资源。
- 自适应停止机制可实时判断共识是否稳定，适合高效评估场景。
随着推理能力提升，大语言模型被广泛用于自动化评判任务。现有方法多采用简单聚合（如多数投票），即使单个代理答案正确，仍可能失效。为此，我们提出多智能体辩论评判框架，让代理协作推理并迭代优化判断。通过数学建模分析代理交互，证明辩论能放大正确性，优于静态集成。为提高效率，引入稳定性检测机制，基于时变的Beta-Binomial混合模型建模评委共识动态，并以分布相似性（柯尔莫戈洛夫-斯米尔诺夫检验）实现自适应终止。在多个基准和模型上的实验表明，该框架在保持计算效率的同时，显著提升了评判准确率。
原文摘要 · Abstract (English)
With advancements in reasoning capabilities, Large Language Models (LLMs) are increasingly employed for automated judgment tasks. While LLMs-as-Judges offer promise in automating evaluations, current approaches often rely on simplistic aggregation methods (e.g., majority voting), which can fail even when individual agents provide correct answers. To address this, we propose a multi-agent debate judge framework where agents collaboratively reason and iteratively refine their responses. We formalize the debate process mathematically, analyzing agent interactions and proving that debate amplifies correctness compared to static ensembles. To enhance efficiency, we introduce a stability detection mechanism that models judge consensus dynamics via a time-varying Beta-Binomial mixture, with adaptive stopping based on distributional similarity (Kolmogorov-Smirnov test). This mechanism models the judges' collective correct rate dynamics using a time-varying mixture of Beta-Binomial distributions and employs an adaptive stopping criterion based on distributional similarity (Kolmogorov-Smirnov statistic). Experiments across multiple benchmarks and models demonstrate that our framework improves judgment accuracy over majority voting while maintaining computational efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。