用多个小模型评估大模型，解决评价循环矛盾问题
Towards Acyclic Preference Evaluation of Language Models via Multiple Evaluators
- 引入多评估者构建偏好图，再融合去噪以消除循环偏好
- 在10个基准上验证，能更准确排序模型、选最优回复、选训练数据
- 小模型组合比大模型更可靠，适合评估与优化大模型
尽管大型语言模型（LLMs）取得显著进展，但其输出质量的偏好评估仍是关键挑战。现有方法通常使用一个强模型作为评判者进行两两比较，但单一评估者易导致循环偏好——即A优于B，B优于C，C又优于A，造成矛盾结果。为此，我们提出PGED（偏好图集成与去噪），利用多个基于模型的评估者构建偏好图，再进行集成与去噪，获得无环、非矛盾的评估结果。理论证明该框架可恢复真实偏好结构。在10个基准上的实验表明，PGED在三项应用中表现优异：模型排序、测试时扩展中的响应选择、模型微调的数据选择。值得注意的是，使用Llama3-8B、Mistral-7B、Qwen2-7B等小模型组合，性能超过Qwen2-72B等大模型，展现出提升评估可靠性与模型性能的有效性。
原文摘要 · Abstract (English)
Despite the remarkable success of Large Language Models (LLMs), evaluating their outputs' quality regarding preference remains a critical challenge. While existing works usually leverage a strong LLM as the judge for comparing LLMs' response pairwisely, such a single-evaluator approach is vulnerable to cyclic preference, i.e., output A is better than B, B than C, but C is better than A, causing contradictory evaluation results. To address this, we introduce PGED (Preference Graph Ensemble and Denoising), a novel approach that leverages multiple model-based evaluators to construct preference graphs, and then ensembles and denoises these graphs for acyclic, non-contradictory evaluation results. We provide theoretical guarantees for our framework, demonstrating its efficacy in recovering the ground truth preference structure. Extensive experiments on ten benchmarks demonstrate PGED's superiority in three applications: 1) model ranking for evaluation, 2) response selection for test-time scaling, and 3) data selection for model fine-tuning. Notably, PGED combines small LLM evaluators (e.g., Llama3-8B, Mistral-7B, Qwen2-7B) to outperform strong ones (e.g., Qwen2-72B), showcasing its effectiveness in enhancing evaluation reliability and improving model performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。