HelaBERT提升僧伽罗语理解，用双池化头增强分类效果。
HelaBERT: Enhancing Sinhala Language Understanding with Dual Pooling Classification Head
- 基于10亿词的僧伽罗语数据训练双版本BERT模型，适配其黏着语特征。
- 双池化头在情感分析上显著提效，新闻分类任务也有适度提升。
- 适合研究僧伽罗语NLP、低资源语言处理的学者与开发者使用。
我们提出HelaBERT，一个从零开始预训练的双版本基于BERT的掩码语言模型，基于约10亿词的僧伽罗语文本，数据来源包括MADLAD-400、CulturaX及自建语料（新闻文章、僧伽罗语维基百科与网络爬取数据）。HelaBERT-Small（约2330万参数，6层）与HelaBERT-Large（约1.1亿参数，12层）均采用针对僧伽罗语黏着形态与复杂文字设计的SentencePiece Unigram分词器（词汇量32,000）。我们在四个下游僧伽罗语文本分类任务上评估：新闻类别分类、新闻源分类、情感分析与写作风格分类，采用5次独立种子运行及分层80/20训练/测试划分。此外，我们提出并系统评估了一种双池化分类头，在情感分析中表现一致提升，对HelaBERT-Small在新闻类别分类中带来适度增益；而标准[CLS]-线性头在新闻源分类（短输入平均长度）上仍具竞争力。两个模型均已发布，以支持僧伽罗语自然语言处理研究。
原文摘要 · Abstract (English)
We present HelaBERT, a family of two BERT-based masked language models pre-trained from scratch on approximately 1 billion tokens of Sinhala text sourced from MADLAD-400, CulturaX, and a custom corpus comprising news articles, Sinhala Wikipedia, and web crawl data. HelaBERT-Small (~23.3M parameters, 6 layers) and HelaBERT-Large (~110M parameters, 12 layers) both use a SentencePiece Unigram tokenizer (vocabulary size 32,000) tailored to Sinhala's agglutinative morphology and complex script. We evaluate both models on four downstream Sinhala text classification tasks: news category classification, news source classification, sentiment analysis, and writing style classification, using 5 independent seed runs with stratified 80/20 train/test splits. We additionally propose a dual pooling classification head and evaluate it systematically across all four tasks, finding consistent improvements on sentiment analysis and a moderate gain on news category classification for HelaBERT-Small, while the standard [CLS]-linear head remains competitive on news source classification, a headline-level task with short average input length. We release both models to support further research in Sinhala NLP.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。