提出稀疏注意力机制，让模型更好理解表格结构。
Structural Deep Encoding for Table Question Answering
- 设计新型稀疏注意力掩码，保留表格行列关系
- 在多个基准上提升准确率，最高达7.2%提升
- 适合需要理解复杂表格的问答任务
尽管基于Transformer的架构在处理文本信息方面表现出色，但其对表格数据的直接应用通常涉及将表格扁平化，这种简化会丢失行、列和单元格之间的关键依赖关系，同时对大表格带来可扩展性挑战。为解决这些问题，已有工作探索了特殊标记、结构化嵌入和稀疏注意力模式。本文对表格编码技术进行了全面分析，揭示了注意力稀疏性在保持表格结构信息中的关键作用。我们还提出一组针对表格数据的新稀疏注意力掩码设计，不仅提升了计算效率，还增强了结构完整性，从而实现更优的整体性能。
原文摘要 · Abstract (English)
Although Transformers-based architectures excel at processing textual information, their naive adaptation for tabular data often involves flattening the table structure. This simplification can lead to the loss of essential inter-dependencies between rows, columns, and cells, while also posing scalability challenges for large tables. To address these issues, prior works have explored special tokens, structured embeddings, and sparse attention patterns. In this paper, we conduct a comprehensive analysis of tabular encoding techniques, which highlights the crucial role of attention sparsity in preserving structural information of tables. We also introduce a set of novel sparse attention mask designs for tabular data, that not only enhance computational efficiency but also preserve structural integrity, leading to better overall performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。