首个面向越南语的文本嵌入评估基准，助力AI模型落地
VN-MTEB: Vietnamese Massive Text Embedding Benchmark
- 用大模型自动翻译并筛选英文数据，生成高质量越南语样本
- 涵盖41个数据集、6类任务，验证大模型在越南语上表现更优
- 适合做多语言NLP、内容安全与推荐系统的研究者使用
越南互联网流量和网络毒性均居世界前列，亟需在推荐与内容管控中部署文本嵌入模型。然而，缺乏大规模、任务多样化的评测数据集，使模型评估困难。为此，我们构建了首个越南语文本嵌入基准VN-MTEB，通过自研自动化框架将大量英文样本从Massive Text Embedding Benchmark翻译而来。利用大语言模型（LLMs）与先进嵌入模型进行翻译与过滤，确保语言自然性、语义保真度，同时保留命名实体识别（NER）和代码片段。该基准包含41个数据集，覆盖六类任务。分析显示，采用旋转位置编码（Rotary Positional Embedding）的更大更复杂模型在嵌入任务中表现优于绝对位置编码模型。数据集已开源至HuggingFace：https://huggingface.co/collections/GreenNode/vn-mteb-68871433f0f7573b8e1a6686。
原文摘要 · Abstract (English)
Vietnam ranks among the top countries in terms of both internet traffic and online toxicity. As a result, implementing embedding models for recommendation and content control duties in applications is crucial. However, a lack of large-scale test datasets, both in volume and task diversity, makes it tricky for scientists to effectively evaluate AI models before deploying them in real-world, large-scale projects. To solve this important problem, we introduce a Vietnamese benchmark, VN-MTEB for embedding models, which we created by translating a large number of English samples from the Massive Text Embedding Benchmark using our new automated framework. We leverage the strengths of large language models (LLMs) and cutting-edge embedding models to conduct translation and filtering processes to retain high-quality samples, guaranteeing a natural flow of language and semantic fidelity while preserving named entity recognition (NER) and code snippets. Our comprehensive benchmark consists of 41 datasets from six tasks specifically designed for Vietnamese text embeddings. In our analysis, we find that bigger and more complex models using Rotary Positional Embedding outperform those using Absolute Positional Embedding in embedding tasks. Datasets are available at HuggingFace: https://huggingface.co/collections/GreenNode/vn-mteb-68871433f0f7573b8e1a6686
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。