非上下文BERT嵌入在马拉地语低资源任务中表现优于FastText。
Non-Contextual BERT or FastText? A Comparative Analysis
- 用BERT第一层嵌入做查表式非上下文表示
- 非上下文BERT在新闻分类等任务上胜过FastText
- 适合资源匮乏语言的快速高效模型部署
低资源语言因缺乏大规模标注数据和语言资源，在自然语言处理中面临严峻挑战。嵌入方法的选择对提升任务性能至关重要。尽管上下文BERT嵌入需完整前向传播，非上下文BERT嵌入仅依赖查表计算。现有研究多聚焦于上下文BERT，对非上下文嵌入探索不足。本研究分析了来自MuRIL和MahaBERT的非上下文BERT嵌入，以及IndicFT和MahaFT的FastText嵌入在马拉地语新闻分类、情感分析和仇恨言论检测中的效果，并与上下文及压缩变体对比。结果表明，从模型第一层提取的非上下文BERT嵌入优于FastText嵌入，为低资源NLP提供了一种有前景的替代方案。
原文摘要 · Abstract (English)
Natural Language Processing (NLP) for low-resource languages, which lack large annotated datasets, faces significant challenges due to limited high-quality data and linguistic resources. The selection of embeddings plays a critical role in achieving strong performance in NLP tasks. While contextual BERT embeddings require a full forward pass, non-contextual BERT embeddings rely only on table lookup. Existing research has primarily focused on contextual BERT embeddings, leaving non-contextual embeddings largely unexplored. In this study, we analyze the effectiveness of non-contextual embeddings from BERT models (MuRIL and MahaBERT) and FastText models (IndicFT and MahaFT) for tasks such as news classification, sentiment analysis, and hate speech detection in one such low-resource language Marathi. We compare these embeddings with their contextual and compressed variants. Our findings indicate that non-contextual BERT embeddings extracted from the model's first embedding layer outperform FastText embeddings, presenting a promising alternative for low-resource NLP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。