用多头注意力提升俄语形态标注，支持未登录词且可在普通显卡训练。
A Multi-head-based architecture for effective morphological tagging in Russian with open dictionary
- 基于多头注意力设计新架构，通过子词向量聚合实现开集词典支持。
- 在SinTagRus和Taiga数据集上，部分词性标注准确率达98%-99%以上。
- 无需预训练、速度快、适合实际部署，尤其适合资源有限场景。
本文提出一种基于多头注意力的新架构，用于解决俄语形态标注问题。词向量预处理包括将词拆分为子词，再通过训练好的方法将子词向量聚合为词向量，从而支持开集词典，并考虑词的组成部分（如前缀、后缀等）进行形态特征分析。该设计使模型未来可处理训练集中未出现的词汇。在SinTagRus和Taiga数据集上的计算实验表明，某些语法类别准确率可达98%-99%以上，优于以往结果。对于九成以上的词，模型能准确预测所有语法范畴，并在不适用时明确标注。该模型可在消费级显卡上训练，保留多头注意力相比RNN的优势，无需大规模无标签文本预训练（如BERT），且处理速度高于以往方法。
原文摘要 · Abstract (English)
The article proposes a new architecture based on Multi-head attention to solve the problem of morphological tagging for the Russian language. The preprocessing of the word vectors includes splitting the words into subtokens, followed by a trained procedure for aggregating the vectors of the subtokens into vectors for tokens. This allows to support an open dictionary and analyze morphological features taking into account parts of words (prefixes, endings, etc.). The open dictionary allows in future to analyze words that are absent in the training dataset. The performed computational experiment on the SinTagRus and Taiga datasets shows that for some grammatical categories the proposed architecture gives accuracy 98-99% and above, which outperforms previously known results. For nine out of ten words, the architecture precisely predicts all grammatical categories and indicates when the categories must not be analyzed for the word. At the same time, the model based on the proposed architecture can be trained on consumer-level graphics accelerators, retains all the advantages of Multi-head attention over RNNs (RNNs are not used in the proposed approach), does not require pretraining on large collections of unlabeled texts (like BERT), and shows higher processing speed than previous results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。