IHUBERT通过语义去重和领域均衡预训练，显著提升波斯语语言模型性能。
IHUBERT: Vector-Based Semantic Deduplication and Domain-Balanced Pretraining for Persian Resources
- 构建45GB高质量波斯语语料库，结合向量数据库实现语义去重与领域平衡。
- 在多个波斯语任务上表现优异，尤其在抽取式问答任务中刷新纪录。
- 专为波斯语设计的分词器有效捕捉词形变化，适合多领域自然语言理解应用。
波斯语预训练语言模型受限于大规模高质量语料稀缺及评估任务单一。本文提出IHUBERT，基于RoBERTa-base架构（125M参数）在45GB精选的Sepahr-Danesh语料子集（约7-8B tokens）上从零训练。为提升语料质量并减少冗余，采用多阶段预处理流程：归一化、精确与近似重复删除、匿名化，以及基于向量数据库的语义去重，以控制跨领域和语体分布平衡。额外在全语料上训练了139k词表大小的BPE分词器，更好捕捉波斯语形态与拼写变异。IHUBERT在七个波斯语自然语言理解基准上评估，涵盖命名实体识别、情感分析、主题分类、自然语言推理、抽取式问答和关系抽取，使用标准指标（实体级F1、Macro-F1、EM/F1）。其在抽取式问答任务中表现最佳，于PQuAD（F1 88.3542）和ParsiNLU-RC（F1 49.0987）上排名第一，在FarsTail任务上取得最高宏平均F1（0.8350）。在命名实体识别与主题分类任务上保持竞争力（如ParsTwiNER F1 0.8308；DigiMag Macro-F1 0.7953），而关系抽取仍为短板（PERLEX Macro-F1 0.6684）。控制变量分词器消融实验表明，在相同词表规模下，BPE相比WordPiece产生更少子词碎片，支持分词设计。总体而言，IHUBERT通过语义优化的大规模预训练和广泛的任务评估，推动波斯语语言建模发展。
原文摘要 · Abstract (English)
Persian pretrained language models (PLMs) are still limited by the scarcity of large-scale, high-quality pretraining corpora and by insufficient evaluation beyond standard classification and NER tasks. We present IHUBERT, a monolingual Persian PLM trained from scratch with the RoBERTa-base encoder (125M parameters) on a 45 GB curated subset of the Sepahr-Danesh collection (about 7-8B tokens). To improve corpus quality and reduce redundancy, we employ a multi-stage preprocessing pipeline that includes normalization, exact and near-duplicate removal, anonymization, and vector-database-based semantic deduplication for distribution balancing control across domains and registers. We additionally train a 139k-vocabulary BPE tokenizer on the full pretraining corpus to better capture Persian morphology and orthographic variation. IHUBERT is evaluated on seven Persian NLU benchmarks covering NER, sentiment analysis, topic classification, NLI, extractive question answering, and relation extraction, using task-standard metrics (entity-level F1, Macro-F1, EM/F1). IHUBERT achieves its strongest gains on extractive QA, ranking first on both PQuAD (F1 88.3542) and ParsiNLU-RC (F1 49.0987), and attains the best result on FarsTail (Macro-F1 0.8350). On NER and topic classification, it remains competitive (e.g., 0.8308 F1 on ParsTwiNER; 0.7953 Macro-F1 on DigiMag), while relation extraction remains the main remaining gap (0.6684 Macro-F1 on PERLEX). A controlled tokenizer ablation on the IHUBERT pretraining corpus shows that BPE yields slightly lower subword fragmentation than WordPiece at matched vocabulary size, supporting our tokenization design. Overall, IHUBERT advances Persian language modeling through semantically curated large-scale pretraining and broad evaluation across both classification and comprehension-oriented tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。