对比大中小模型生成新闻摘要的连贯性，发现所有模型均优于人工参考摘要。
Consistency Evaluation of News Article Summaries Generated by Large (and Small) Language Models
- 用多种模型生成摘要，通过传统指标与大模型评估结合
- 在XL-Sum数据集上，所有模型摘要一致性均高于参考摘要
- 提出元评估分数，直接衡量大模型评估系统本身的性能
文本摘要是自然语言处理中的关键任务，应用涵盖信息检索与内容生成。尽管大型语言模型（LLMs）在生成流畅的抽象摘要方面表现优异，但可能产生脱离原文的幻觉细节。无论采用何种生成方法，高质量的自动化评估仍是一个开放问题。本文探索了包括TextRank、BART、Mistral-7B-Instruct和OpenAI GPT-3.5-Turbo在内的多种技术生成摘要，并使用传统指标如ROUGE和BERTScore，以及基于大模型的评估方法——直接评估摘要与原文的一致性。我们引入一种元评估分数，用于直接衡量大模型评估系统（提示+模型）的表现。结果表明，在XL-Sum数据集上，所有摘要生成模型产生的摘要一致性均超过参考摘要。
原文摘要 · Abstract (English)
Text summarizing is a critical Natural Language Processing (NLP) task with applications ranging from information retrieval to content generation. Large Language Models (LLMs) have shown remarkable promise in generating fluent abstractive summaries but they can produce hallucinated details not grounded in the source text. Regardless of the method of generating a summary, high quality automated evaluations remain an open area of investigation. This paper embarks on an exploration of text summarization with a diverse set of techniques, including TextRank, BART, Mistral-7B-Instruct, and OpenAI GPT-3.5-Turbo. The generated summaries are evaluated using traditional metrics such as the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) Score and Bidirectional Encoder Representations from Transformers (BERT) Score, as well as LLM-powered evaluation methods that directly assess a generated summary's consistency with the source text. We introduce a meta evaluation score which directly assesses the performance of the LLM evaluation system (prompt + model). We find that that all summarization models produce consistent summaries when tested on the XL-Sum dataset, exceeding the consistency of the reference summaries.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。