arXiv:2409.18193cs.CL2024-09NAACL被引 8

为87种低资源语言构建了高效静态词向量库,性能媲美大模型。

GrEmLIn: A Repository of Green Baseline Embeddings for 87 Low-Resource Languages Injected with Multilingual Graph Knowledge

  • 用多语言图知识增强GloVe,生成无需参数的静态嵌入
  • 在词汇相似度任务上超越E5等上下文嵌入,差距仅5-10%
  • 适合资源有限场景,尤其适用于低资源语言处理

基于大语言模型(LLMs)的上下文嵌入在多种语言中可用,但对低资源语言覆盖不足。使用LLMs处理这些语言成本高昂,训练与推理均需大量计算资源。静态词嵌入更高效(“绿色”),对极低资源语言仍有价值。然而,缺乏针对多样化语言的综合性静态嵌入库。为此,我们提出GrEmLIn,一个集中式仓库,包含87种中低资源语言的绿色静态基线嵌入。我们采用新方法,通过整合多语言图知识增强GloVe嵌入,使静态嵌入在推理时无参数,同时性能媲美上下文嵌入。实验表明,GrEmLIn在词汇相似度任务上优于E5等先进上下文嵌入;在情感分析、自然语言推断等外在任务中,平均性能差距仅为5%-10%或更小,前提是目标任务词汇重叠充分;仅在主题分类任务上表现较差。代码与嵌入已公开于https://huggingface.co/DFKI。

原文摘要 · Abstract (English)

Contextualized embeddings based on large language models (LLMs) are available for various languages, but their coverage is often limited for lower resourced languages. Using LLMs for such languages is often difficult due to a high computational cost; not only during training, but also during inference. Static word embeddings are much more resource-efficient ("green"), and thus still provide value, particularly for very low-resource languages. There is, however, a notable lack of comprehensive repositories with such embeddings for diverse languages. To address this gap, we present GrEmLIn, a centralized repository of green, static baseline embeddings for 87 mid- and low-resource languages. We compute GrEmLIn embeddings with a novel method that enhances GloVe embeddings by integrating multilingual graph knowledge, which makes our static embeddings competitive with LLM representations, while being parameter-free at inference time. Our experiments demonstrate that GrEmLIn embeddings outperform state-of-the-art contextualized embeddings from E5 on the task of lexical similarity. They remain competitive in extrinsic evaluation tasks like sentiment analysis and natural language inference, with average performance gaps of just 5-10\% or less compared to state-of-the-art models, given a sufficient vocabulary overlap with the target task, and underperform only on topic classification. Our code and embeddings are publicly available at https://huggingface.co/DFKI.

词向量低资源语言静态嵌入多语言图

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。