arXiv:2606.15474cs.AIstat.AP2026-06被引 3

解决大模型评估中系统与评分模型漂移的混淆问题,实现精准归因。

Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines

论文配图:Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines
图 1 · 摘自论文原文
  • 用人类标注的锚点集定期检验评分模型,识别是系统变差还是评分标准变了。
  • 在两次真实变更中,60次全检出评分模型漂移,零误判为系统问题。
  • 成本仅为全程强评估的64%,适合持续监控生产环境的大模型服务。

大模型产品持续评估依赖一个被视为基准的廉价评分模型:它实时打分,一旦分数下降就触发警报。但该评分模型本身也是通过API调用的模型,其版本更新或评分提示调整会改变打分逻辑,导致每次警报都难以分辨是系统性能下降还是评分标准变化。本文提出一种解决方案:固定一组人类标注的锚点集,由当前评分模型周期性重评;结合第二层赌注型过程(e-process)检测评分模型与人类判断之间的差距,并采用守门窗口规则判定结果为{无、系统、评分模型}。理论证明了任意时间有效性、单向识别性(仅评分模型可移动锚点)、归因竞赛机制(锚点必须比主进程跑得快),以及过程正交性。在两个真实场景中,对一次无声版本升级,60次实验全部正确检测到评分模型漂移且无误判;对一次严格提示污染,120次中有110次正确归因(守门窗宽300)。而行业默认的滚动z检验在75%无漂移流中误报。所有实验在第二个领域(摘要生成)复现,无需调参,差异符合预期预测:严格提示变更在此领域影响更大,锚点更快触发,归因完美(240/240)。监测成本约为全程强评估的0.64倍,或在低质量但便宜的模式下为0.21。

原文摘要 · Abstract (English)

Continuous evaluation of LLM products relies on a strong LLM judge treated as ground truth: a cheap monitor scores every interaction and a team is paged when the score drifts down. But the judge is itself a model behind an API, and a silent version bump or scoring-prompt update changes how it scores -- so every drift alarm is ambiguous between a worse product and a changed judge. We resolve the ambiguity with a fixed, human-labeled anchor set that the current judge re-scores at a steady interleave, a second betting e-process on the judge-versus-human gap, and a guard-window rule returning a verdict in {none, system, judge}. We prove anytime-validity, one-way identification (only the judge can move the anchors), an attribution race whose design law is that the anchors must out-run the main process they guard, and process orthogonality. On two real judge changes, a silent version bump is detected as judge drift in 60/60 runs with zero judge-to-system misattribution, and a contaminating strict-prompt change is correctly attributed on 110 of 120 runs at guard width 300 -- while the industry-default rolling z-test false-alarms on 75% of drift-free streams. Every experiment replicates on a second domain (TL;DR summarization) with nothing re-tuned, and where the domains differ the differences are the ones the race predicts: the strict-prompt change shifts scores harder there, so the anchors fire faster and attribution becomes perfect (240/240). The monitor runs at approximately 0.64 of the cost of strong-judging every item, or 0.21 in a cheaper-but-deafer regime.

大模型评估漂移检测归因分析持续监控

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。