arXiv:2603.20975cs.CLcs.LG2026-03被引 2

通过分析大模型代理间的分歧结构,提升复杂推理任务的不确定性量化精度。

DiscoUQ: Structured Disagreement Analysis for Uncertainty Quantification in LLM Agent Ensembles

  • 基于语言特征与嵌入几何结构,挖掘代理间分歧的深层语义信息。
  • 在4个基准上平均AUROC达0.802,校准误差仅0.036,优于基线。
  • 特别擅长处理模糊的弱分歧场景,适合高可靠性推理系统应用。

多代理大语言模型系统在复杂推理任务中日益普及,但现有不确定性量化方法仅依赖浅层投票统计,忽略代理推理中的丰富语义信息。本文提出DiscoUQ框架,通过提取并利用代理间分歧的结构特征——包括语言属性(证据重叠、论证强度、分歧深度)与嵌入几何(聚类距离、离散度、凝聚力)——生成校准良好的置信度估计。提出三种渐进复杂的方法:DiscoUQ-LLM(基于大模型提取特征的逻辑回归)、DiscoUQ-Embed(基于嵌入几何的逻辑回归)和DiscoUQ-Learn(融合全部特征的神经网络)。在四个不同基准(StrategyQA、MMLU、TruthfulQA、ARC-Challenge)上,使用5个代理的Qwen3.5-27B系统进行评估,DiscoUQ-LLM平均AUROC达0.802,优于最佳基线(LLM Aggregator,0.791),且校准更优(ECE 0.036 vs. 0.098)。学习到的特征跨基准泛化能力强,性能下降接近零,在简单投票失效的‘弱分歧’区域表现提升最显著。

原文摘要 · Abstract (English)

Multi-agent LLM systems, where multiple prompted instances of a language model independently answer questions, are increasingly used for complex reasoning tasks. However, existing methods for quantifying the uncertainty of their collective outputs rely on shallow voting statistics that discard the rich semantic information in agents' reasoning. We introduce DiscoUQ, a framework that extracts and leverages the structure of inter-agent disagreement -- both linguistic properties (evidence overlap, argument strength, divergence depth) and embedding geometry (cluster distances, dispersion, cohesion) -- to produce well-calibrated confidence estimates. We propose three methods of increasing complexity: DiscoUQ-LLM (logistic regression on LLM-extracted structure features), DiscoUQ-Embed (logistic regression on embedding geometry), and DiscoUQ-Learn (a neural network combining all features). Evaluated on four diverse benchmarks (StrategyQA, MMLU, TruthfulQA, ARC-Challenge) with a 5-agent system using Qwen3.5-27B, DiscoUQ-LLM achieves an average AUROC of 0.802, outperforming the best baseline (LLM Aggregator, 0.791) while being substantially better calibrated (ECE 0.036 vs. 0.098). The learned features generalize across benchmarks with near-zero performance degradation and provide the largest improvements where they are most needed: in the ambiguous "weak disagreement" tier where simple vote counting fails.

不确定性量化大模型代理分歧分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。