用双向Mamba模型提升阿拉伯语长文本理解效率，效果媲美主流Transformer。
AraSSM: A bidirectional state-space encoder for Arabic masked language modeling
- 基于Mamba的双向状态空间模型，线性计算复杂度解决注意力机制瓶颈。
- 在4个阿拉伯语基准上表现接近或超过Transformer基线，尤其在情感分类达96.37%准确率。
- 仅用消费级显卡训练，适合资源有限的研究者快速复现与应用。
预训练的Transformer编码器如AraBERT、MARBERT和CAMeLBERT已成为阿拉伯语自然语言理解的标准骨干，但其自注意力机制随序列长度呈二次增长，限制了长文档处理效率。Mamba是一种选择性状态空间模型（SSM），可实现线性时间序列建模，是注意力机制的有力替代方案，然而目前尚无专门针对阿拉伯语预训练的双向Mamba编码器。本文提出AraSSM，一个通过掩码语言建模在阿拉伯语维基百科与CulturaX文本组合语料上预训练的双向Mamba编码器，全程在四块消费级NVIDIA RTX 2080Ti GPU（11GB）上训练，耗时约十天。我们在四个主流阿拉伯语自然语言理解任务上评估AraSSM：情感分类（HARD）、命名实体识别（ANERcorp）、抽取式问答（ARCD）和自然语言推理（XNLI-ar），采用AraBERT提出的每任务评估协议，并报告三次微调种子下的均值±标准差。AraSSM在情感分类上达到96.37±0.03%准确率，匹配或超越已发布的基础尺寸Transformer基线；在抽取式问答（ARCD：32.19±1.07 EM，63.79±0.25 F1）和命名实体识别（ANERcorp：81.54±0.30实体级F1）上表现竞争力；在自然语言推理（XNLI-ar：72.83±0.07%准确率）上略逊于基础尺寸Transformer，但所有结果均为从零开始在消费级硬件上训练所得。
原文摘要 · Abstract (English)
Pretrained Transformer encoders such as AraBERT, MARBERT, and CAMeLBERT have become the standard backbone for Arabic natural language understanding, but their self-attention mechanism scales quadratically with sequence length, which limits efficiency on long documents. Mamba, a selective state-space model (SSM), offers linear-time sequence modeling as a competitive alternative to attention, yet no dedicated bidirectional Mamba encoder pretrained specifically for Arabic currently exists. We introduce AraSSM, a bidirectional Mamba encoder pretrained via masked language modeling on a corpus combining Arabic Wikipedia and CulturaX text, trained end-to-end on four consumer-grade NVIDIA RTX 2080Ti GPUs (11GB) over approximately ten days. We evaluate AraSSM by fine-tuning on four established Arabic NLU benchmarks covering sentiment classification (HARD), named entity recognition (ANERcorp), extractive question answering (ARCD), and natural language inference (XNLI-ar), following the per-task evaluation protocol introduced by AraBERT, and report results as mean +/- standard deviation across three fine-tuning seeds. AraSSM matches or exceeds published base-sized Transformer baselines on sentiment classification (96.37 +/- 0.03% accuracy on HARD), is competitive on extractive QA (32.19 +/- 1.07 EM, 63.79 +/- 0.25 F1 on ARCD) and named entity recognition (81.54 +/- 0.30 entity-level F1 on ANERcorp), and trails the base-sized Transformer range on natural language inference (72.83 +/- 0.07% accuracy on XNLI-ar), despite being trained entirely from scratch on consumer hardware rather than large-scale accelerator clusters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。