用极坐标设计注意力机制，让模型更懂文档布局。
DocPolarBERT: A Pre-trained Model for Document Understanding with Relative Polar Coordinate Encoding of Layout Structures
- 用相对极坐标替代笛卡尔坐标编码文本块位置。
- 在数据量不足IIT-CDIP六分之一时仍达顶尖效果。
- 适合需要高效布局理解的文档智能场景。
我们提出DocPolarBERT，一种面向文档理解的布局感知BERT模型，无需使用绝对2D位置嵌入。通过将自注意力机制扩展为基于相对极坐标系的文本块位置编码，而非传统的笛卡尔坐标系。尽管预训练数据集规模不足广泛使用的IIT-CDIP语料库的六分之一，该模型仍达到当前最优性能。结果表明，精心设计的注意力机制可有效弥补预训练数据量的不足，为文档理解提供了一种高效且有效的替代方案。
原文摘要 · Abstract (English)
We introduce DocPolarBERT, a layout-aware BERT model for document understanding that eliminates the need for absolute 2D positional embeddings. We extend self-attention to take into account text block positions in relative polar coordinate system rather than the Cartesian one. Despite being pre-trained on a dataset more than six times smaller than the widely used IIT-CDIP corpus, DocPolarBERT achieves state-of-the-art results. These results demonstrate that a carefully designed attention mechanism can compensate for reduced pre-training data, offering an efficient and effective alternative for document understanding.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。