用Transformer模型检测孟加拉语语法错误，提升自动打字助手准确性
Bangla Grammatical Error Detection Leveraging Transformer-based Token Classification
- 将语法错误检测转为分词分类任务，结合Transformer模型
- 在2.5万条文本上测试，最佳模型Levenshtein距离得分1.04
- 融合多模型输出并规则后处理，适合语言工具开发者参考
孟加拉语是全球使用人数第七多的语言，但其自动化语法检查系统仍研究不足。语法错误检测旨在识别孟加拉语文本中存在语法、标点或拼写错误的子串，对构建自动孟加拉语输入助手至关重要。本文将该任务转化为分词分类问题，采用先进的基于Transformer的模型，并融合多个模型输出，通过规则后处理生成更可靠、全面的结果。系统在包含超过2.5万条文本的多源数据集上进行评估，最佳模型在Levenshtein距离指标上达到1.04。文章还对系统各组件进行了详细分析。
原文摘要 · Abstract (English)
Bangla is the seventh most spoken language by a total number of speakers in the world, and yet the development of an automated grammar checker in this language is an understudied problem. Bangla grammatical error detection is a task of detecting sub-strings of a Bangla text that contain grammatical, punctuation, or spelling errors, which is crucial for developing an automated Bangla typing assistant. Our approach involves breaking down the task as a token classification problem and utilizing state-of-the-art transformer-based models. Finally, we combine the output of these models and apply rule-based post-processing to generate a more reliable and comprehensive result. Our system is evaluated on a dataset consisting of over 25,000 texts from various sources. Our best model achieves a Levenshtein distance score of 1.04. Finally, we provide a detailed analysis of different components of our system.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。