用内容单元评估摘要质量，提升小模型摘要效果
SCURank: Ranking Multiple Candidate Summaries with Summary Content Units for Enhanced Summarization
- 基于摘要内容单元（SCU）衡量信息丰富度与语义重要性
- 在多个数据集上优于传统指标和大模型排名方法
- 适合需要高质量摘要蒸馏的开发者和研究者
小语言模型（SLMs）如BART可通过知识蒸馏达到与大语言模型（LLMs）相当的摘要性能。然而，现有基于LLM的摘要候选排名策略存在不稳定性，而经典指标（如ROUGE）又难以区分高质量摘要。为此，我们提出SCURank框架，通过引入摘要内容单元（SCUs）来增强摘要生成。该方法不依赖不稳定的对比或表面重叠，而是基于信息内容的丰富度与语义重要性进行评估。我们验证了SCURank在从多个多样化LLMs中蒸馏摘要时的有效性。实验结果表明，SCURank在多种评价指标和数据集上均优于传统指标和基于LLM的排名方法。此外，研究发现融合多样化LLM摘要能提升模型抽象能力与整体蒸馏性能，证实了以信息为中心的排名在多LLM蒸馏中的优势。代码已开源：https://github.com/IKMLab/SCURank。
原文摘要 · Abstract (English)
Small language models (SLMs), such as BART, can achieve summarization performance comparable to large language models (LLMs) via distillation. However, existing LLM-based ranking strategies for summary candidates suffer from instability, while classical metrics (e.g., ROUGE) are insufficient to rank high-quality summaries. To address these issues, we introduce \textbf{SCURank}, a framework that enhances summarization by leveraging \textbf{Summary Content Units (SCUs)}. Instead of relying on unstable comparisons or surface-level overlap, SCURank evaluates summaries based on the richness and semantic importance of information content. We investigate the effectiveness of SCURank in distilling summaries from multiple diverse LLMs. Experimental results demonstrate that SCURank outperforms traditional metrics and LLM-based ranking methods across evaluation measures and datasets. Furthermore, our findings show that incorporating diverse LLM summaries enhances model abstractiveness and overall distilled model performance, validating the benefits of information-centric ranking in multi-LLM distillation. The code for SCURank is available at https://github.com/IKMLab/SCURank.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。