为印地语、孟加拉语和马拉地语构建统一命名实体识别模型
TriNER: A Series of Named Entity Recognition Models For Hindi, Bengali & Marathi
- 基于自定义Transformer与预训练模型微调，实现多语言统一建模
- 在6类实体上达到92.11的F1分数，显著提升跨语言一致性
- 适合需要多语言实体识别的印度本土NLP应用开发者
印度丰富的文化和语言多样性给自然语言处理（NLP）带来了诸多挑战，尤其是在命名实体识别（NER）领域。NER旨在识别并分类文本中的实体类型，如人名、地点、组织、数字等，对上下文感知的匿名化等下游任务具有重要意义。本文致力于为印度三大主要语言——印地语、孟加拉语和马拉地语构建一个多语言NER模型。我们训练了一个定制的Transformer模型，并对多个预训练模型进行了微调，在总计6个实体类别上取得了92.11的F1分数。通过本工作，我们提出一种统一模型以执行NER，显著减少了三种语言间实体类别和标签名称的不一致问题。
原文摘要 · Abstract (English)
India's rich cultural and linguistic diversity poses various challenges in the domain of Natural Language Processing (NLP), particularly in Named Entity Recognition (NER). NER is a NLP task that aims to identify and classify tokens into different entity groups like Person, Location, Organization, Number, etc. This makes NER very useful for downstream tasks like context-aware anonymization. This paper details our work to build a multilingual NER model for the three most spoken languages in India - Hindi, Bengali & Marathi. We train a custom transformer model and fine tune a few pretrained models, achieving an F1 Score of 92.11 for a total of 6 entity groups. Through this paper, we aim to introduce a single model to perform NER and significantly reduce the inconsistencies in entity groups and tag names, across the three languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。