将文档结构融入注意力机制，提升语言模型的抽象理解能力。
StructFormer: Document Structure-based Masked Attention and its Impact on Language Model Pre-Training
- 基于文档结构设计掩码注意力，替代传统全局注意力
- 在arXiv数据上预训练后，模型在文档理解任务中表现更优
- 适合关注长文本建模与文档级理解的研究者
当前主流语言模型多基于Transformer架构及其注意力机制，但序列变长时计算开销呈指数增长，限制了其处理长文本的能力。近期研究尝试通过局部与全局注意力等稀疏机制缓解此问题。尽管这类稀疏注意力在理论上具备图灵完备性，其对预训练的实际影响尚未明确。本研究通过arXiv数据构建结构感知文本语料，并与纯文本对照组进行对比预训练，分析注意力模式变化及其对下游任务的影响。结果表明，融入文档结构信息可显著提升模型在抽象任务（如文档理解）中的表现。
原文摘要 · Abstract (English)
Most state-of-the-art techniques for Language Models (LMs) today rely on transformer-based architectures and their ubiquitous attention mechanism. However, the exponential growth in computational requirements with longer input sequences confines Transformers to handling short passages. Recent efforts have aimed to address this limitation by introducing selective attention mechanisms, notably local and global attention. While sparse attention mechanisms, akin to full attention in being Turing-complete, have been theoretically established, their practical impact on pre-training remains unexplored. This study focuses on empirically assessing the influence of global attention on BERT pre-training. The primary steps involve creating an extensive corpus of structure-aware text through arXiv data, alongside a text-only counterpart. We carry out pre-training on these two datasets, investigate shifts in attention patterns, and assess their implications for downstream tasks. Our analysis underscores the significance of incorporating document structure into LM models, demonstrating their capacity to excel in more abstract tasks, such as document understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。