对比17个大模型在多维度下的摘要表现，找出了各模型的优劣与适用场景。
An Empirical Comparison of Text Summarization: A Multi-Dimensional Evaluation of Large Language Models
- 构建多维度评估框架，涵盖事实一致性、语义相似度等指标。
- Gemini系列在效率上表现最优，Claude-3在人类感知质量上最佳。
- 技术类文本难总结，50词最准，150词最像人写，需权衡取舍。
文本摘要对缓解新闻、医疗、商业等领域的信息过载至关重要。本研究采用新颖的多维度评估框架，对17个大型语言模型（OpenAI、Google、Anthropic及开源模型）在七种不同数据集（BigPatent、BillSum、CNN/DailyMail、PubMed、SAMSum、WikiHow、XSum）上，以50、100、150三个输出长度进行评估，使用事实一致性、语义相似度、词汇重叠率和人类质量评分等指标，并考虑效率因素。结果显示模型性能差异显著：deepseek-v3在事实准确性上领先，claude-3-5-sonnet在人类感知质量上最佳，gemini-1.5-flash与gemini-2.0-flash在处理效率与成本效益方面表现突出。模型表现随数据集变化巨大，在技术类领域表现较差，但在对话类内容上效果良好。发现事实一致性在50词时最佳，而感知质量在150词时最高，存在显著权衡。研究为高风险应用（需准确）与资源受限场景（需高效）提供基于证据的选择建议。该方法通过整合质量与运营考量，明确揭示了准确性、效率与成本间的权衡，助力特定场景下的模型选型。
原文摘要 · Abstract (English)
Text summarization is crucial for mitigating information overload across domains like journalism, medicine, and business. This research evaluates summarization performance across 17 large language models (OpenAI, Google, Anthropic, open-source) using a novel multi-dimensional framework. We assessed models on seven diverse datasets (BigPatent, BillSum, CNN/DailyMail, PubMed, SAMSum, WikiHow, XSum) at three output lengths (50, 100, 150 tokens) using metrics for factual consistency, semantic similarity, lexical overlap, and human-like quality, while also considering efficiency factors. Our findings reveal significant performance differences, with specific models excelling in factual accuracy (deepseek-v3), human-like quality (claude-3-5-sonnet), and processing efficiency/cost-effectiveness (gemini-1.5-flash, gemini-2.0-flash). Performance varies dramatically by dataset, with models struggling on technical domains but performing well on conversational content. We identified a critical tension between factual consistency (best at 50 tokens) and perceived quality (best at 150 tokens). Our analysis provides evidence-based recommendations for different use cases, from high-stakes applications requiring factual accuracy to resource-constrained environments needing efficient processing. This comprehensive approach enhances evaluation methodology by integrating quality metrics with operational considerations, incorporating trade-offs between accuracy, efficiency, and cost-effectiveness to guide model selection for specific applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。