针对超长文档摘要，提出分段再合并方法提升低资源场景效果
A Split-then-Join Approach to Abstractive Summarization for Very Long Documents in a Low Resource Setting
- 将长文档拆分成4096词元片段，分段处理后再合并结果
- 在2万词元以上文档上微调模型，显著提升超长文本摘要能力
- 适用于低资源环境下需要处理超长文档的摘要任务
现有最先进的摘要模型BIGBIRD-PEGASUS最大支持4096词元，导致超长文档摘要性能下降。常规做法是截断文档，但会丢失信息。本文提出一种新方法：先筛选长度超过20,000词元的文档，利用预训练的BIGBIRD-PEGASUS模型，在其他领域数据集上进行微调，并通过将文档-摘要对分块的方式扩充数据集，使每段适配4096词元限制，从而缓解领域偏移和过拟合问题。该方法有效提升了超长文档摘要性能。源代码见https://github.com/lhfazry/SPIN-summ。
原文摘要 · Abstract (English)
$\texttt{BIGBIRD-PEGASUS}$ model achieves $\textit{state-of-the-art}$ on abstractive text summarization for long documents. However it's capacity still limited to maximum of $4,096$ tokens, thus caused performance degradation on summarization for very long documents. Common method to deal with the issue is to truncate the documents. In this reasearch, we'll use different approach. We'll use the pretrained $\texttt{BIGBIRD-PEGASUS}$ model by fine tuned the model on other domain dataset. First, we filter out all documents which length less than $20,000$ tokens to focus on very long documents. To prevent domain shifting problem and overfitting on transfer learning due to small dataset, we augment the dataset by splitting document-summary training pair into parts, to fit the document into $4,096$ tokens. Source code available on $\href{https://github.com/lhfazry/SPIN-summ}{https://github.com/lhfazry/SPIN-summ}$.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。