用大模型自我评估改进摘要质量，无需微调。
LLM-ReSum: A Framework for LLM Reflective Summarization through Self-Evaluation
- 让大模型自评摘要并循环优化，形成闭环反馈。
- 在三类文档中提升事实准确率33%、覆盖度39%。
- 适合需要高质量摘要的法律、科研等专业领域。
大语言模型生成摘要的可靠评估仍是开放挑战，尤其在异构领域和不同文档长度下。我们对14种自动摘要评估指标及基于大模型的评估器进行了全面元评估，覆盖七个数据集，涵盖新闻、科学、政府和法律等五类文档，文本长度从2K到27K词不等，包含超过1500条人工标注摘要。结果表明，传统词汇重叠指标（如ROUGE、BLEU）与人类判断相关性弱或为负，而任务特定神经指标和基于大模型的评估器在语言质量评估上显著更优。基于此，我们提出LLM-ReSum框架，通过无微调的大模型自评估与生成闭环反馈，改进摘要质量。在三类领域中，低质量摘要的事实准确率提升达33%，覆盖度提升39%，人工评估者在89%情况下更偏好优化后摘要。此外，我们引入PatentSumEval，一个包含180条专家评估摘要的新基准，用于专利文档摘要评估。所有代码与数据集将开源于GitHub。
原文摘要 · Abstract (English)
Reliable evaluation of large language model (LLM)-generated summaries remains an open challenge, particularly across heterogeneous domains and document lengths. We conduct a comprehensive meta-evaluation of 14 automatic summarization metrics and LLM-based evaluators across seven datasets spanning five domains, covering documents from short news articles to long scientific, governmental, and legal texts (2K-27K words) with over 1,500 human-annotated summaries. Our results show that traditional lexical overlap metrics (e.g., ROUGE, BLEU) exhibit weak or negative correlation with human judgments, while task-specific neural metrics and LLM-based evaluators achieve substantially higher alignment, especially for linguistic quality assessment. Leveraging these findings, we propose LLM-ReSum, a self-reflective summarization framework that integrates LLM-based evaluation and generation in a closed feedback loop without model finetuning. Across three domains, LLM-ReSum improves low-quality summaries by up to 33% in factual accuracy and 39% in coverage, with human evaluators preferring refined summaries in 89% of cases. We additionally introduce PatentSumEval, a new human-annotated benchmark for legal document summarization comprising 180 expert-evaluated summaries. All code and datasets will be released in GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。