用小模型实现多语言文本生成评估，比大模型更稳定高效
Beyond LLM-as-a-Judge: Deterministic Metrics for Multilingual Generative Text Evaluation
- 用107种语言的合成数据训练小型确定性评分模型
- 在6种语言的问答、翻译、摘要任务中表现接近大模型
- 适合需要快速、可复现评估的多语言研究者
尽管大型语言模型（LLM）被广泛用于自动评估生成文本，但其输出成本高，对提示设计、语言和聚合策略敏感，严重限制了可复现性。为此，我们提出 extbf{OmniScore}，一组基于小参数量（<1B）模型的互补性、确定性学习度量。OmniScore 近似 LLM 判官行为，同时保持传统模型评分的低延迟与一致性。我们使用大规模合成监督数据（约564,000条实例，涵盖107种语言）训练模型，并在8,617条人工标注实例上进行评估。OmniScore 支持多种场景下的可靠、多维度评分，包括参考式、源基式和混合评估。我们在6种语言的问答（QA）、翻译和摘要任务中进行了评估。结果表明，轻量级、确定性的学习度量为前沿大模型提供了一种高度实用且可扩展的替代方案。模型与数据集可于 https://huggingface.co/collections/QCRI/omniscore 获取。
原文摘要 · Abstract (English)
While Large Language Models (LLMs) are increasingly adopted as automated judges for evaluating generated text, their outputs are often costly, and highly sensitive to prompt design, language, and aggregation strategies, severely, which limits reproducibility. To address these challenges, we propose \textbf{\textit{OmniScore}}, a family of complementary, deterministic learned metrics developed using small size ($<$1B) parameter models. OmniScore approximates LLM-judge behavior while preserving the low latency and consistency of traditional model-based scoring. We trained the models large-scale synthetic supervision ($\sim$564k instances, in \textbf{107 languages}) and evaluated using 8,617 manually annotated instances. The OmniScore family supports reliable, multi-dimensional scores across a variety of settings, including reference-based, source-grounded, and hybrid evaluations. We evaluate these models across question answering (QA), translation, and summarization in \textbf{6 languages}. Our results demonstrate that lightweight, deterministic learned metrics provide a highly practical and scalable alternative to frontier LLMs. Our models and datasets can be found at https://huggingface.co/collections/QCRI/omniscore
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。