提出可证明无偏的LLM评判框架，保障评估结果不受隐藏偏差影响。
Towards Provably Unbiased LLM Judges via Bias-Bounded Evaluation
- 设计平均偏差有界性算法，量化并限制评判模型的偏差影响
- 在Arena-Hard-Auto上实现tau=0.5、delta=0.01的严格偏差约束
- 保持61-99%原始排序相关性，适合高可靠性评估场景
随着AI系统从简单对话走向复杂自主工作流，其依赖自动化、可验证反馈机制的趋势日益明显。在真实标签稀缺或非确定性场景中，以LLM作为评判者（LLM-as-a-Judge）成为可行方案。然而现有研究尚未提供能在未知或对抗性偏差下仍具强保障的评估体系。为此，本文提出平均偏差有界性（A-BB）算法框架，形式化保证任何可测量偏差下的危害降低。在Arena-Hard-Auto数据集上，使用4个LLM评判者进行测试，实现tau=0.5、delta=0.01的偏差有界保障，在格式与结构偏差设置下仍保持61%-99%的原始排名相关性，多数评判组合超过80%。代码已开源：https://github.com/penfever/bias-bounded-evaluation。
原文摘要 · Abstract (English)
As AI models progress beyond simple chatbots into more complex workflows, we draw ever closer to the event horizon beyond which AI systems will be utilized in autonomous, self-maintaining feedback loops. Any autonomous AI system will depend on automated, verifiable rewards and feedback; in settings where ground truth is sparse or non-deterministic, one practical source of such rewards is an LLM-as-a-Judge. Although LLM judges continue to improve, the literature has yet to introduce systems capable of enforcing standards with strong guarantees, particularly when bias vectors are unknown or adversarially discovered. To remedy this issue, we propose average bias-boundedness (A-BB), an algorithmic framework which formally guarantees reductions of harm/impact as a result of any measurable bias in an LLM judge. Evaluating on Arena-Hard-Auto with four LLM judges, we achieve (tau=0.5, delta=0.01) bias-bounded guarantees while retaining 61-99% correlation with original rankings across formatting and schematic bias settings, with most judge-bias combinations exceeding 80%. The code to reproduce our findings is available at https://github.com/penfever/bias-bounded-evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。