对比大模型读完整书和凭记忆写摘要，发现后者有时更优。
Large Language Models and Book Summarization: Reading or Remembering, Which Is Better?
- 用大模型基于原文或内部知识生成书籍摘要
- 多数情况读全文得更详细，但部分书记忆摘要更优
- 适合研究大模型长文本理解能力的学者参考
摘要生成是自然语言处理的核心任务。随着大语言模型（LLMs）的发展和上下文窗口扩展至数百万个词元，单次提示即可处理整本书。同时，对于知名书籍，模型仅依靠训练中习得的内部知识也可生成摘要。这引发关键问题：基于内部知识的摘要与基于全文的摘要相比如何？先验知识是否影响模型在输入全文时的摘要表现？本文对前沿大模型的书籍摘要能力进行实验评估，比较了两种方式生成的摘要：(i) 仅依赖模型内部知识；(ii) 使用全书文本。结果表明，一般情况下使用全文能生成更详细的摘要，但某些书籍中内部知识生成的摘要得分更高。这一发现质疑了模型处理长文本摘要的能力——训练中学到的信息在某些情况下甚至优于对全文的摘要。
原文摘要 · Abstract (English)
Summarization is a core task in Natural Language Processing (NLP). Recent advances in Large Language Models (LLMs) and the introduction of large context windows reaching millions of tokens make it possible to process entire books in a single prompt. At the same time, for well-known books, LLMs can generate summaries based only on internal knowledge acquired during training. This raises several important questions: How do summaries generated from internal memory compare to those derived from the full text? Does prior knowledge influence summaries even when the model is given the book as input? In this work, we conduct an experimental evaluation of book summarization with state-of-the-art LLMs. We compare summaries of well-known books produced using (i) only the internal knowledge of the model and (ii) the full text of the book. The results show that having the full text provides more detailed summaries in general, but some books have better scores for the internal knowledge summaries. This puts into question the capabilities of models to perform summarization of long texts, as information learned during training can outperform summarization of the full text in some cases.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。