检测大模型评判时的隐性依赖,避免错误共识误导评估结果。
A Statistical Framework for Auditing Behavioral Dependence and Induced Bias in LLM Judges

- 用信息论指标分析多个大模型在判断任务中的行为耦合
- 发现18个模型存在显著行为依赖,导致评分偏差
- 可用来优化模型集成,提升评测准确率和精度
大语言模型生态系统快速发展,但共享预训练数据、蒸馏与对齐流程可能导致模型间隐藏的行为依赖,即潜在耦合,破坏如‘大模型作为裁判’等多模型系统中独立信号的假设。实践中表现为推理模式相关、故障同步,表面一致实为共同出错。为此,我们提出一种统计审计框架,通过双信息论指标:(i) 难度加权行为耦合指数(BEI),放大简单任务上的同步失败;(ii) 累积信息增益(CIG),捕捉错误响应的方向性对齐。在六个模型族共18个大模型上实验,发现显著行为依赖。该依赖与独立MMLU-Pro测试集上的裁判过度认可偏差正相关(BEI: rho=0.508, p<0.01; CIG: rho=0.520, p<0.01),且在MATH-500基准上同样显著(BEI: rho=0.441, p<0.05; CIG: rho=0.457, p<0.05),表明依赖结构具有跨基准泛化能力。最后,通过去耦合验证器集成重加权,相较多数投票分别实现3.5%和2.6%的准确率与精确率提升。
原文摘要 · Abstract (English)
The rapid growth of the large language model (LLM) ecosystem raises a critical question: are seemingly diverse models truly independent? Shared pretraining data, distillation, and alignment pipelines can induce hidden behavioral dependencies, or latent entanglement, that undermine multi-model systems such as LLM-as-a-judge pipelines and ensemble verification, which implicitly assume independent signals. In practice, this manifests as correlated reasoning patterns and synchronized failures, where apparent agreement reflects shared error modes rather than independent validation. To address this, we develop a statistical framework for auditing behavioral entanglement among black-box LLMs. Our approach introduces a multi-resolution hierarchy that characterizes the joint failure manifold through two information-theoretic metrics: (i) a Difficulty-Weighted Behavioral Entanglement Index (BEI), which amplifies synchronized failures on easy tasks, and (ii) a Cumulative Information Gain (CIG) metric, which captures directional alignment in erroneous responses. Through experiments on 18 LLMs from six model families, we identify statistically significant behavioral entanglement. Such behavioral dependence is associated with judge over-endorsement bias on a disjoint MMLU-Pro evaluation set (rho = 0.508 for BEI and rho = 0.520 for CIG; p < 0.01). The association further transfers to the MATH-500 benchmark (rho = 0.441 for BEI and rho = 0.457 for CIG; p < 0.05), providing cross-benchmark evidence that the identified dependency structure generalizes beyond the data and response format used for its estimation. Finally, we demonstrate a practical use case of entanglement through de-entangled verifier ensemble reweighting, achieving 3.5 and 2.6 percentage-point gains in accuracy and precision, respectively, over majority voting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。