直接用字符生成词向量，提升低资源复杂形态语言的表示效果。
Beyond Subtokens: A Rich Character Embedding for Low-resource and Morphologically Complex Languages
- 基于Transformer构建字符级词向量，融合语义与句法信息。
- 在少量数据下，奇数项识别与前K名排序任务上超越传统方法。
- 适合低资源、形态丰富的语言研究者使用。
基于分词和子词的模型如word2vec、BERT和GPTs是自然语言处理的主流方法。但这些方法在输入表示上存在局限，难以充分捕捉拼写相似性和形态变化，尤其在高度屈折且资源匮乏的语言中表现不佳。为此，我们提出一种直接从字符串计算词向量的方法，结合语义与句法信息，称为富字符嵌入（Rich Character Embeddings, RCE）。同时，设计了一种融合Transformer与卷积机制的混合模型。两种向量可作为现有架构中字典或子词嵌入的即插即用替代方案，在大模型（如BERT）和小模型（如word2vec）中均有潜力提升性能。我们在多种任务上进行评估：SWAG、屈折预测、隐喻与回文检测等。实验表明，在有限数据条件下，使用OddOneOut和TopK指标，RCE显著优于传统分词方法。
原文摘要 · Abstract (English)
Tokenization and sub-tokenization based models like word2vec, BERT and the GPTs are the state-of-the-art in natural language processing. Typically, these approaches have limitations with respect to their input representation. They fail to fully capture orthographic similarities and morphological variations, especially in highly inflected and under-resource languages. To mitigate this problem, we propose to computes word vectors directly from character strings, integrating both semantic and syntactic information. We denote this transformer-based approach Rich Character Embeddings (RCE). Furthermore, we propose a hybrid model that combines transformer and convolutional mechanisms. Both vector representations can be used as a drop-in replacement for dictionary- and subtoken-based word embeddings in existing model architectures. It has the potential to improve performance for both large context-based language models like BERT and small models like word2vec for under-resourced and morphologically rich languages. We evaluate our approach on various tasks like the SWAG, declension prediction for inflected languages, metaphor and chiasmus detection for various languages. Our experiments show that it outperforms traditional token-based approaches on limited data using OddOneOut and TopK metrics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。