首个针对卢旺达语表格数据的预训练模型，提升低资源语言信息提取能力。
TabuLM: Morphology-Aware Tabular Pre-training for Low-Resource Languages
- 在卢旺达语表格中引入行、列和单元格类型嵌入，增强结构感知
- 在526组问答上达到62.0%准确率，领先基线11.7-12.7个百分点
- 适合低资源语言表格理解与本地化应用研究者使用
我们提出TabuLM，首个在卢旺达语表格数据上预训练的语言模型。卢旺达语是卢旺达超过1200万人使用的形态丰富的班图语，但缺乏专用表格表征学习资源。TabuLM扩展了KinyaBERT-large（双层形态变换器），引入加性行、列和单元格类型嵌入，以及可学习的表格结构注意力偏置，强化同行与同列注意力。预训练采用两项新目标：掩码单元恢复（MCR），掩蔽整个单元格并要求从行列上下文中重建；列类型预测（CTP），从观测到的单元格值预测列语义类型。模型在来自NISR、RAB、REB和MoH开放数据门户的172张卢旺达政府表格（约35,000个单元格）上进行预训练，并构建了首个原生卢旺达语表格问答基准TabQA-kin，包含526个问答对，覆盖31张表格和四种问题类型。TabuLM在TabQA-kin上取得62.0%精确匹配率，较KinyaBERT-large提升5.7个百分点，较多语言基线（mBERT 49.3%，XLM-R 50.0%）高出11.7-12.7个百分点。分析表明，表格结构嵌入对比较和查找类问题最为关键，而形态感知提供互补增益。代码、数据及预训练检查点已公开。
原文摘要 · Abstract (English)
We present TabuLM, the first language model pre-trained on Kinyarwanda tabular data. Kinyarwanda is a morphologically rich Bantu language spoken by over 12 million people in Rwanda, yet lacks any dedicated tabular representation learning resource. TabuLM extends KinyaBERT-large, a two-tier morphological transformer, with additive row, column, and cell-type embeddings and a learned table-structure attention bias that sharpens same-row and same-column attention. Pre-training uses two new objectives: Masked Cell Recovery (MCR), which masks entire cells and forces reconstruction from row and column context, and Column Type Prediction (CTP), which predicts column semantic types from observed cell values. We pre-train on 172 Rwandan government tables (~35,000 cells) from NISR, RAB, REB, and MoH open-data portals, and introduce TabQA-kin, the first native Kinyarwanda table question-answering benchmark comprising 526 QA pairs across 31 tables and four question types. TabuLM achieves 62.0% exact match on TabQA-kin, outperforming KinyaBERT-large by 5.7 EM points and all multilingual baselines (mBERT 49.3%, XLM-R 50.0%) by 11.7-12.7 points. Analysis shows that structural table embeddings are most decisive for comparison and lookup questions, while morphological awareness provides complementary gains. Our code, data, and pre-trained checkpoint are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。