arXiv:2607.28636cs.CLcs.CY2026-07

用不同模型交叉审计大模型判断,提升判别公正性。

Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges

论文配图:Chain-of-Models: Cross-Model Auditing for Bias-Robust LLM Judges
图 1 · 摘自论文原文
  • 让第二个模型审查第一个模型的推理过程再做判断。
  • 不同模型对不同偏见类型有更好纠错效果,最优审计器具偏见特异性。
  • 按偏见类型自动选最佳审计模型,准确率显著高于固定模型或无审计。

大语言模型日益充当自动化评判者,但其判断仍易受认知偏见影响。现有缓解方法多依赖提示驱动去偏,对多种偏见脆弱;或依赖人工评估,难以扩展。本文研究跨模型审计机制「链式模型」(Chain-of-Models, CoM),即由第二个模型审查首个模型的推理轨迹后生成最终判断。核心问题是:审计模型应为同一模型、同家族模型,还是异家族模型?在6个模型家族中的9个模型、4种认知偏见和4个事实数据集上,我们发现审计模型身份在两方面起关键作用:第一,独立抗偏能力不预示审计有效性——Kimi-K2.5在多个偏见上表现最强,却无法有效纠正Qwen2.5-72B的偏见推理;第二,最优审计器具有偏见特异性:GPT-4o在从众、权威、分心偏见上最强,GLM-5在谄媚偏见上最强。基于此,我们提出一种按偏见类型选择审计器的规则,综合功能多样性、单偏见抗性及校准后的审计效果进行评分。在训练/测试划分下,该选择器在四类偏见样本上的准确率达0.884,显著优于最强固定审计器(0.824)和无审计基线(0.805)。相关数据、配置与LLM代理技能已公开于https://anonymous.4open.science/r/chain-of-models-B585。

原文摘要 · Abstract (English)

LLMs increasingly serve as automated judges, but their judgments remain vulnerable to cognitive biases. Existing mitigations mostly rely on prompt-driven debiasing, which is brittle across bias types, or human evaluation, which does not scale. We study \emph{Chain-of-Models} (CoM), an automated audit pipeline in which a second model inspects the first model's reasoning trace before producing the final judgment. The key design question is whether the auditor should be the same model, a same-family model, or a different-family model. Across 9 models from 6 families, 4 cognitive biases, and 4 factual datasets, we find that auditor identity matters in two ways. First, standalone bias resistance does not predict audit effectiveness: Kimi-K2.5 is the strongest standalone model on several biases, yet is a weak auditor for Qwen2.5-72B's biased traces. Second, the best auditor is bias-specific: GPT-4o is strongest on bandwagon, authority, and distraction, while GLM-5 is strongest on sycophancy. We operationalize these findings with a per-bias auditor selection rule that, given the bias type, scores candidates along functional diversity, per-bias standalone resistance, and calibrated audit effectiveness. Under a calibration/test split, the selector reaches the highest accuracy across the four biased slices ($0.884$ vs.\ $0.824$ for the strongest single fixed auditor and $0.805$ for the no-audit baseline). We release data, configurations, and an LLM-agent skill at https://anonymous.4open.science/r/chain-of-models-B585 .

大模型评测偏见检测审计机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。