LLM裁判偏爱风格而非事实，导致评测失真。
Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking
- 构建SOS-Bench基准，测试真实对齐效果
- 发现LLM裁判偏好华丽表达，忽略安全与知识
- 微调阶段比偏好优化更重要，数据多样性是关键
2022年11月ChatGPT发布后，后训练方法迅速兴起，多数声称通过更好匹配人类成对偏好实现更优对齐，常以LLM裁判打分作为依据。本文探究：LLM裁判偏好是否真正反映其他具体对齐指标的进步？我们定义了具体的对齐度量标准，提出SOS-Bench（Substance Outweighs Style Benchmark），据我们所知是当前最大、最标准化、可复现的LLM元基准。结果表明：(1) LLM裁判偏好与安全、世界知识、指令遵循等实际指标无相关性；(2) LLM裁判存在强烈隐含偏见，优先考虑风格而非事实性与安全性；(3) 后训练中监督微调（SFT）阶段对对齐影响最大，而非偏好优化（PO）阶段，数据规模与提示多样性是主要驱动因素。代码与完整结果详见https://github.com/penfever/sos-bench。
原文摘要 · Abstract (English)
The release of ChatGPT in November 2022 sparked an explosion of interest in post-training and an avalanche of new preference optimization (PO) methods. These methods claim superior alignment by virtue of better correspondence with human pairwise preferences, often measured by LLM-judges. In this work, we attempt to answer the following question -- do LLM-judge preferences translate to progress on other, more concrete metrics for alignment, and if not, why not? We define a concrete metric for alignment, and introduce SOS-Bench (Substance Outweighs Style Benchmark), which is to the best of our knowledge the largest standardized, reproducible LLM meta-benchmark to date. We find that (1) LLM-judge preferences do not correlate with concrete measures of safety, world knowledge, and instruction following; (2) LLM-judges have powerful implicit biases, prioritizing style over factuality and safety; and (3) the supervised fine-tuning (SFT) stage of post-training, and not the PO stage, has the greatest impact on alignment, with data scaling and prompt diversity as the driving factors. Our codebase and complete results can be found at https://github.com/penfever/sos-bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。