arXiv:2606.10716cs.CLcs.AI2026-06

通过扩展注意力范围,让模型更好提取长文档关键词。

Attention Expansion: Enhancing Keyphrase Extraction from Long Documents with Attention-Augmented Contextualized Embeddings

论文配图:Attention Expansion: Enhancing Keyphrase Extraction from Long Documents with Attention-Augmented Contextualized Embeddings
图 1 · 摘自论文原文
  • 用预训练词向量补充上下文信息,扩大模型感知范围。
  • 在五个数据集上提升关键短语抽取的准确率,最高F1提升显著。
  • 适用于各类主流模型,尤其适合长文档且计算成本低。

预训练语言模型在关键短语抽取任务中表现优异,主要得益于其生成丰富上下文表征的能力。然而,长文档的关键证据可能分散在遥远段落中,受限于模型上下文窗口,难以联合捕捉。尽管长上下文大模型可处理更广文本,但其高计算开销限制了高效高吞吐量的关键词抽取应用。为此,本文提出注意力扩展机制,利用预训练词嵌入将文档外上下文块的信息融入模型的词元表示,从而在不需全文档注意力或昂贵大模型推理的前提下,扩展基于PLM模型的有效上下文范围。我们在五种不同类型的PLM骨干网络(通用、科学、任务专用及长上下文编码器)上,采用两种训练方式,在五个来自科学与新闻领域的基准数据集上评估该方法。实验结果表明,注意力扩展在所有设置下均持续提升性能,优于现有最先进模型,显著提高F1分数。改进效果涵盖领域特定、任务专用及原生长上下文模型,说明该机制提供互补信息而非仅弥补输入长度限制。这些结果确立了注意力扩展作为长文档关键词抽取的一种高效且有效策略。

原文摘要 · Abstract (English)

Pre-trained language models (PLMs) have achieved strong performance in keyphrase extraction (KPE), largely due to their ability to generate rich contextualized representations. However, long-document KPE remains challenging because salient keyphrase evidence may be scattered across distant document sections that cannot be jointly captured within the limited context window of most PLMs. Although long-context large language models (LLMs) can process broader textual contexts, their computational cost limits their practicality for efficient and high-throughput KPE. To overcome this limitation, we propose an attention expansion mechanism that augments PLM token representations with information from surrounding out-of-context chunks using pre-trained word embeddings. The proposed mechanism expands the effective contextual scope of PLM-based KPE models without requiring full-document attention or expensive LLM-based inference. We evaluate our approach across five PLM backbones, including general-purpose, scientific, task-specific, and long-context encoders, using two training regimes and five benchmark corpora from scientific and news domains. Experimental results demonstrate that attention expansion consistently enhances KPE performance across all evaluation settings, outperforming state-of-the-art models and yielding notable improvements in F1 score. The improvements extend to domain-specific, task-specialized, and native long-context models, showing that the proposed mechanism provides complementary information rather than merely compensating for limited input length. These results establish attention expansion as an efficient and effective strategy for long-document KPE.

关键词抽取长文档注意力机制预训练模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。