arXiv:2601.07245cs.AIcs.CL2026-01

用多个模型回答的共识,提升大模型推理可靠性。

Learning to Trust the Crowd: A Multi-Model Consensus Reasoning Engine for Large Language Models

  • 融合多个异构模型输出,通过结构化特征与图神经网络做决策
  • 在多个数据集上准确率比最强单模型高4.6个百分点,比多数投票高8.1点
  • 适合追求高可靠性的实际部署场景,尤其资源受限环境

大语言模型虽平均表现强,但个体实例中常出现幻觉、脆性失败和置信度不准确问题。本文从多模型共识视角出发,研究如何从多个异构大模型的回答中判断哪个最可能正确。提出一种多模型共识推理引擎,将多个模型输出作为输入,通过语义嵌入、成对相似性、聚类统计、词汇与结构线索、推理质量评分、置信度估计及模型先验等特征构建结构化表示,再利用梯度提升树、列表排序和基于答案相似性图的图神经网络进行判断。在GSM8K、ARC-Challenge、HellaSwag和TruthfulQA的紧凑子集上,使用三个开源权重的LLM进行实验,最佳图注意力模型在宏平均准确率上比最强单模型提高4.6个百分点,比多数投票提升8.1个百分点,同时降低Brier分数并减少TruthfulQA中的幻觉现象。消融分析表明,语义一致性和聚类特征影响最大,推理质量与模型先验提供互补增益,证明监督式多模型共识是实现更可靠大模型行为的可行路径,即使在单机有限资源下亦可实现。

原文摘要 · Abstract (English)

Large language models (LLMs) achieve strong average performance yet remain unreliable at the instance level, with frequent hallucinations, brittle failures, and poorly calibrated confidence. We study reliability through the lens of multi-model consensus: given responses from several heterogeneous LLMs, can we learn which answer is most likely correct for a given query? We introduce a Multi-Model Consensus Reasoning Engine that treats the set of LLM outputs as input to a supervised meta-learner. The system maps natural language responses into structured features using semantic embeddings, pairwise similarity and clustering statistics, lexical and structural cues, reasoning-quality scores, confidence estimates, and model-specific priors, and then applies gradient-boosted trees, listwise ranking, and graph neural networks over similarity graphs of answers. Using three open-weight LLMs evaluated on compact, resource-constrained subsets of GSM8K, ARC-Challenge, HellaSwag, and TruthfulQA, our best graph-attention-based consensus model improves macro-average accuracy by 4.6 percentage points over the strongest single LLM and by 8.1 points over majority vote, while also yielding lower Brier scores and fewer TruthfulQA hallucinations. Ablation and feature-importance analyses show that semantic agreement and clustering features are most influential, with reasoning-quality and model-prior features providing complementary gains, suggesting supervised multi-model consensus is a practical route toward more reliable LLM behavior, even in a modest single-machine setup.

大模型可靠性多模型共识推理增强

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。