用自动拆解事实的方法，让大模型回答评估更透明可解释。
Chatbot Arena Meets Nuggets: Towards Explanations and Diagnostics in the Evaluation of LLM Responses
- 自动将长答案拆解为关键事实点，实现可解释评估
- 7000场对抗评测中，事实得分与人类偏好高度相关
- 适合需要诊断模型回答缺陷的研究者和开发者
对抗评测（battles）通过人机偏好对比已成为评估大模型输出质量的主流方法，最近也被扩展至检索增强生成（RAG）系统。然而，这类方法在复杂信息查询场景下缺乏解释性和诊断性。为此，新兴的nugget评估法将长篇回答分解为原子事实，突出“优质回答”所需的关键信息。本文利用AutoNuggetizer框架，对LMArena提供的约7000场Search Arena对抗数据进行全自动分析。结果表明，nugget得分与人类偏好存在显著相关性，验证了该方法在可解释、可诊断评估中的潜力。所有复现代码已开源：https://github.com/castorini/lmsys_nuggetize。
原文摘要 · Abstract (English)
Battles, or side-by-side comparisons in so-called arenas that elicit human preferences, have emerged as a popular approach for assessing the output quality of LLMs. Recently, this idea has been extended to retrieval-augmented generation (RAG) systems. While undoubtedly representing an advance in evaluation, battles have at least two drawbacks, particularly in the context of complex information-seeking queries: they are neither explanatory nor diagnostic. Recently, the nugget evaluation methodology has emerged as a promising approach to evaluate the quality of RAG answers. Nuggets decompose long-form LLM-generated answers into atomic facts, highlighting important pieces of information necessary in a "good" response. In this work, we apply our AutoNuggetizer framework to analyze data from roughly 7K Search Arena battles provided by LMArena in a fully automatic manner. Our results show a significant correlation between nugget scores and human preferences, showcasing promise in our approach to explainable and diagnostic system evaluations. All the code necessary to reproduce results in our work is available in https://github.com/castorini/lmsys_nuggetize.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。