arXiv:2603.25450cs.AI2026-03被引 2

用多个模型互评答案,识别大模型自信犯错的情况。

Cross-Model Disagreement as a Label-Free Correctness Signal

  • 让一个模型判断另一个模型的答案是否合理,无需重新生成。
  • 在多个任务上准确率比传统置信度指标高,MMLU上提升至0.75的AUROC。
  • 适合部署监控、模型路由和数据过滤,无需额外训练。

在无正确标签的情况下检测语言模型错误是安全部署的核心挑战。现有方法依赖模型自身的不确定性(如词元熵或置信度),但在最危险的“自信错误”场景下失效——模型虽错误却高度确定。本文提出跨模型分歧作为正确性信号:给定一个模型的输出答案,通过另一个验证模型单次前向传播计算其对答案的惊讶程度或不确定性,无需生成新内容,也无需标注。我们将其具体化为交叉模型困惑度(CMP)和交叉模型熵(CME)。两者在涵盖推理、检索和数学解题的任务上(MMLU、TriviaQA、GSM8K)均优于内部不确定性基线。在MMLU上,CMP的平均AUROC达到0.75,显著高于内部熵的0.59。该方法为无监督正确性评估提供了可直接集成的、无需训练的实用方案,适用于部署监控、模型路由、选择性预测、数据过滤及大规模生产系统监督。

原文摘要 · Abstract (English)

Detecting when a language model is wrong without ground truth labels is a fundamental challenge for safe deployment. Existing approaches rely on a model's own uncertainty -- such as token entropy or confidence scores -- but these signals fail critically on the most dangerous failure mode: confident errors, where a model is wrong but certain. In this work we introduce cross-model disagreement as a correctness indicator -- a simple, training-free signal that can be dropped into existing production systems, routing pipelines, and deployment monitoring infrastructure without modification. Given a model's generated answer, cross-model disagreement computes how surprised or uncertain a second verifier model is when reading that answer via a single forward pass. No generation from the verifying model is required, and no correctness labels are needed. We instantiate this principle as Cross-Model Perplexity (CMP), which measures the verifying model's surprise at the generating model's answer tokens, and Cross-Model Entropy (CME), which measures the verifying model's uncertainty at those positions. Both CMP and CME outperform within-model uncertainty baselines across benchmarks spanning reasoning, retrieval, and mathematical problem solving (MMLU, TriviaQA, and GSM8K). On MMLU, CMP achieves a mean AUROC of 0.75 against a within-model entropy baseline of 0.59. These results establish cross-model disagreement as a practical, training-free approach to label-free correctness estimation, with direct applications in deployment monitoring, model routing, selective prediction, data filtering, and scalable oversight of production language model systems.

大模型评测无监督评估模型协作部署监控

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。