用大模型生成数据，让小语种也能拥有高质量词向量。
Bootstrapping Embeddings for Low Resource Languages
- 用适配器组合和跨语言微调生成合成三元组数据。
- 在多语言任务上性能接近真实数据训练的基准模型。
- 适合资源匮乏语言的嵌入模型构建，可大规模推广。
词向量模型对现代自然语言处理至关重要，但高性能模型依赖精心构建的监督微调数据。高资源语言如英语有充足数据，但数百种其他语言却缺乏此类数据。我们研究大语言模型是否能弥合这一差距。测试了三种生成合成三元组数据以优化嵌入模型的方法：上下文学习，以及两种新方法——适配器组合与跨语言微调的LLM生成器（XL-LoRA）。结果表明，尽管上下文学习仍不及非合成基线，但适配器组合和XL-LoRA在多种任务与语言中均取得显著性能提升，为多种语言提供了一条清晰且可扩展的高性能嵌入模型构建路径。
原文摘要 · Abstract (English)
Embedding models are crucial to modern NLP. However, the creation of the most effective models relies on carefully constructed supervised finetuning data. For high resource languages, such as English, such datasets are readily available. However, for hundreds of other languages, they are simply non-existent. We investigate whether the advent of large language models can help to bridge this gap. We test three different strategies for generating synthetic triplet data used to optimise embedding models. These include in-context learning as well as two novel approaches, leveraging adapter composition and cross lingual finetuning of the LLM generator (XL-LoRA) respectively. We find that while in-context learning still falls short of strong non-synthetic baselines, adapter composition and XL-LoRA yield strong performance gains across a wide array of tasks and languages, offering a clear, scalable pathway to producing performant embedding models for a wide variety of languages.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。