构建多语言RAG评估基准，用小模型模拟大模型打分。
MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems
- 用启发式指标训练代理裁判，替代昂贵的大模型评分
- 18种语言上与GPT-4o打分相关性达0.909（Kendall Tau）
- 适合关注多语言生成与高效评估的研究者
传统检索增强生成（RAG）评估依赖启发式指标，需人工偏好作为标准；而竞技场式评估虽更可靠，却需昂贵的大语言模型（LLM）作裁判。本文提出一种高效方法：利用启发式指标训练一个代理裁判，输出类大模型的评判结果。我们构建了MIRAGE-Bench，一个针对维基百科、覆盖18种语言的合成竞技场式RAG评估基准，用于多语言答案生成评测。该基准深度融合启发式特征与大模型裁判。我们在19个多语言LLM上进行测试，发现代理裁判与GPT-4o作为教师在Bradley-Terry框架下相关性高达0.909（Kendall Tau）。结果显示，当前专有及大型开源模型在该基准上占据主导地位。代码与数据集已公开：https://github.com/vectara/mirage-bench。
原文摘要 · Abstract (English)
Traditional retrieval-augmented generation (RAG) benchmarks evaluate systems using heuristic-based metrics, but these require human preferences as the ground truth for reference. In contrast, arena-based benchmarks, where systems compete against each other, require an expensive large language model (LLM) as a judge for a reliable evaluation. We present a simple efficient technique to combine the best of both worlds. The idea is to train a surrogate judge using heuristic metrics as input, to output the LLM as a judge prediction. In our work, we develop MIRAGE-Bench, a synthetic arena-based RAG benchmark for 18 diverse languages on Wikipedia focused on multilingual answer generation evaluation. It extensively couples both heuristic features and LLM as a judge for evaluation. We benchmark 19 multilingual LLMs, and observe a high correlation (Kendall Tau ($τ$) = 0.909) using our surrogate judge and between GPT-4o as a teacher using the Bradley-Terry framework. Our results show proprietary and large open-source LLMs currently dominate on MIRAGE-Bench. Our code and datasets are made publicly available here: https://github.com/vectara/mirage-bench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。