arXiv:2603.05314cs.CLcs.AI2026-03被引 2

构建首个大规模波斯语标点恢复数据集,用轻量BERT模型实现高效精准修复。

PersianPunc: A Large-Scale Dataset and BERT-Based Approach for Persian Punctuation Restoration

  • 将标点恢复转为词元级序列标注任务,微调ParsBERT模型。
  • 在1700万样本上达91.33%宏平均F1,优于大模型且更高效。
  • 适合低资源、形态丰富的语言研究,可推广至其他类似语言。

标点恢复对提升自动语音识别(ASR)输出的可读性和下游应用价值至关重要,但波斯语领域仍缺乏系统研究。本文提出PersianPunc,一个包含1700万样本的高质量波斯语标点恢复大规模数据集,通过系统性聚合与过滤现有文本资源构建而成。将标点恢复建模为词元级序列标注任务,微调ParsBERT模型取得优异性能。对比实验表明,尽管大语言模型可完成该任务,但存在过度修正问题(引入非标点修改),且计算开销显著。本文提出的轻量级BERT方法在测试集上达到91.33%的宏平均F1,同时满足实时应用效率要求。数据集与模型已公开(https://huggingface.co/datasets/MohammadJRanjbar/persian-punctuation-restoration 及 https://huggingface.co/MohammadJRanjbar/parsbert-persian-punctuation),旨在推动波斯语NLP发展,并为其他形态丰富、资源匮乏的语言提供可扩展框架。

原文摘要 · Abstract (English)

Punctuation restoration is essential for improving the readability and downstream utility of automatic speech recognition (ASR) outputs, yet remains underexplored for Persian despite its importance. We introduce PersianPunc, a large-scale, high-quality dataset of 17 million samples for Persian punctuation restoration, constructed through systematic aggregation and filtering of existing textual resources. We formulate punctuation restoration as a token-level sequence labeling task and fine-tune ParsBERT to achieve strong performance. Through comparative evaluation, we demonstrate that while large language models can perform punctuation restoration, they suffer from critical limitations: over-correction tendencies that introduce undesired edits beyond punctuation insertion (particularly problematic for speech-to-text pipelines) and substantially higher computational requirements. Our lightweight BERT-based approach achieves a macro-averaged F1 score of 91.33% on our test set while maintaining efficiency suitable for real-time applications. We make our dataset (https://huggingface.co/datasets/MohammadJRanjbar/persian-punctuation-restoration) and model (https://huggingface.co/MohammadJRanjbar/parsbert-persian-punctuation) publicly available to facilitate future research in Persian NLP and provide a scalable framework applicable to other morphologically rich, low-resource languages.

标点恢复波斯语BERT低资源

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。