用问答评估书本的特定方面摘要，发现长文本下RAG更优。
BookAsSumQA: An Evaluation Framework for Aspect-Based Book Summarization via Question Answering
- 基于叙事知识图谱自动生成针对书本的问答对
- 长文本中RAG方法问答准确率显著高于LLM
- 适合研究书籍摘要与大模型评估的学者
面向特定方面的摘要旨在生成突出文本特定维度的摘要，实现更个性化和精准的摘要。然而，由于长文本难以构建参考摘要，该方法在书籍领域的应用尚未被探索。为此，我们提出BookAsSumQA——一种基于问答的书籍特定方面摘要评估框架。该框架通过从叙事知识图谱自动生成特定方面的问题-答案对，依据摘要的问答表现来评估其质量。实验表明，尽管基于大语言模型的方法在短文本上表现更好，但随着文档长度增加，基于检索增强生成（RAG）的方法逐渐展现出更高效率和实用性，更适合用于书籍层面的特定方面摘要。
原文摘要 · Abstract (English)
Aspect-based summarization aims to generate summaries that highlight specific aspects of a text, enabling more personalized and targeted summaries. However, its application to books remains unexplored due to the difficulty of constructing reference summaries for long text. To address this challenge, we propose BookAsSumQA, a QA-based evaluation framework for aspect-based book summarization. BookAsSumQA automatically generates aspect-specific QA pairs from a narrative knowledge graph to evaluate summary quality based on its question-answering performance. Our experiments using BookAsSumQA revealed that while LLM-based approaches showed higher accuracy on shorter texts, RAG-based methods become more effective as document length increases, making them more efficient and practical for aspect-based book summarization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。