用对话语气和模型评分,看清辩论中谁在说谎、谁更可信。
The Confident Liar: Diagnosing Multi-Agent Debate with Log-Probabilities and LLM-as-Judge

- 用双代理辩论架构+大模型评分,量化推理质量。
- 构造者自信度比审计者更能预测推理好坏,检错能力更强。
- 适合研究模型推理可靠性或对抗性测试的人看。
多智能体辩论系统通常只关注最终答案是否正确,忽略了辩论本应产生的中间推理质量。本文研究三种信号的关系:推理词元的逐词对数概率分布、大模型作为裁判对这些词元的评分(遵循指令、论证质量、证据支撑),以及最终任务准确率。我们检验内部置信度信号是否能预测外部评估的推理质量,以及这些信号是否与任务正确性一致。实验涵盖基于评分标准、数学推理和事实问答三个领域。框架采用两代理结构——构造者与审计者,搭配大模型裁判,分别评分每方的指令遵循、理由质量与证据支撑,并标记关键失败。在评分域实验发现稳定的四阶段置信度轨迹,且角色功能存在显著不对称:构造者的置信度与评判推理质量的相关性是审计者的两倍;基于置信度检测关键推理失误的可靠性也更高(构造者AUROC 0.804,审计者0.634)。这些发现推动了本文提出的跨领域深入分析。
原文摘要 · Abstract (English)
Multi-agent debate systems are typically evaluated only on whether the final answer is correct, overlooking the quality of the intermediate reasoning that debate is designed to produce. This paper studies the relationship between three signals in multi-agent debate: token-level log-probability distributions over reasoning tokens, LLM-as-judge rubric scores assigned to those tokens, and final task accuracy. We examine whether internal confidence signals predict externally evaluated reasoning quality, and whether either signal aligns with task correctness, across three domains: rubric-based scoring, mathematical reasoning, and factual question answering. Our framework pairs a two-agent debate architecture -- a Constructor and an Auditor -- with an LLM-as-judge that scores each agent's reasoning along instruction following, justification quality, and evidence grounding, together with a critical-failure flag. Experiments in the rubric-scoring domain reveal a consistent four-phase confidence trajectory and a substantial role asymmetry: confidence aligns with judged reasoning quality roughly twice as strongly for the Constructor as for the Auditor, and confidence-based detection of critical reasoning failures is markedly more reliable for the Constructor (AUROC 0.804) than for the Auditor (0.634). These findings motivate the broader cross-domain investigation proposed in this paper.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。