用大模型生成对抗样本，测试多语言嵌入模型的跨语言语义分辨能力。
Examining Multilingual Embedding Models Cross-Lingually Through LLM-Generated Adversarial Examples
- 基于平行句和大模型生成语义误导但字面相似的干扰项
- 英语中转微调模型在检索任务中表现更好，直接跨语言模型更优
- 揭示不同模型对语言扰动的敏感差异，适合评估跨语言语义理解
现有跨语言语义搜索模型的评估多依赖信息检索与语义文本相似性等任务的数据集。本文提出轻量级评估任务跨语言语义区分（CLSD），仅需平行句和大语言模型（LLM）即可生成对抗性干扰项。CLSD衡量嵌入模型将真实平行句排在语义误导但词汇相似的候选项之上的能力。以新闻领域德语-法语为例构建了CLSD数据集。实验表明，针对检索任务微调的模型通过英语中转获益，而双语语料挖掘模型在直接跨语言设置下表现最佳。细粒度相似性分析显示，不同嵌入模型对语言扰动的敏感性存在差异。代码与数据集已开源，许可协议为AGPL-3.0：https://github.com/impresso/cross_lingual_semantic_discrimination
原文摘要 · Abstract (English)
The evaluation of cross-lingual semantic search models is often limited to existing datasets from tasks such as information retrieval and semantic textual similarity. We introduce Cross-Lingual Semantic Discrimination (CLSD), a lightweight evaluation task that requires only parallel sentences and a Large Language Model (LLM) to generate adversarial distractors. CLSD measures an embedding model's ability to rank the true parallel sentence above semantically misleading but lexically similar alternatives. As a case study, we construct CLSD datasets for German--French in the news domain. Our experiments show that models fine-tuned for retrieval tasks benefit from pivoting through English, whereas bitext mining models perform best in direct cross-lingual settings. A fine-grained similarity analysis further reveals that embedding models differ in their sensitivity to linguistic perturbations. We release our code and datasets under AGPL-3.0: https://github.com/impresso/cross_lingual_semantic_discrimination
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。