用老方法NLI+简单匹配，低成本实现高精度问答评估
Revisiting NLI: Towards Cost-Effective and Human-Aligned Metrics for Evaluating LLMs in Question Answering
- 用现成NLI模型加词汇匹配标记，轻量高效评估LLM答案
- 在长文本问答上达到89.9%准确率，接近GPT-4o表现
- 提出新基准DIVER-QA，适合研究评估指标的学者使用
评估顶尖大语言模型生成的答案颇具挑战：词汇类指标忽略语义细节，而以LLM为裁判的评分方式计算成本高昂。本文重新评估一种轻量级替代方案——基于现成自然语言推理（NLI）评分，并引入简单词汇匹配标志，发现该已有二十年的技术在长文本问答任务中可达到与GPT-4o相当的89.9%准确率，且所需参数量级远低于后者。为严谨测试这些指标的人类对齐性，我们构建了DIVER-QA，一个包含3000个样本的人工标注基准，覆盖五个问答数据集和五种候选大模型。结果表明，低成本NLI评估仍具竞争力，并公开提供DIVER-QA供未来研究使用。
原文摘要 · Abstract (English)
Evaluating answers from state-of-the-art large language models (LLMs) is challenging: lexical metrics miss semantic nuances, whereas "LLM-as-Judge" scoring is computationally expensive. We re-evaluate a lightweight alternative -- off-the-shelf Natural Language Inference (NLI) scoring augmented by a simple lexical-match flag and find that this decades-old technique matches GPT-4o's accuracy (89.9%) on long-form QA, while requiring orders-of-magnitude fewer parameters. To test human alignment of these metrics rigorously, we introduce DIVER-QA, a new 3000-sample human-annotated benchmark spanning five QA datasets and five candidate LLMs. Our results highlight that inexpensive NLI-based evaluation remains competitive and offer DIVER-QA as an open resource for future metric research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。