用大模型和CLIP提升多语言习语图像匹配能力
UoR-NCL at SemEval-2025 Task 1: Using Generative LLMs and CLIP Models for Multilingual Multimodal Idiomaticity Representation
- 用生成式LLM解析习语含义，再用多语言CLIP编码为图像检索表示
- 融合多模态表征后，在英葡双语任务上显著优于原始词组表征
- 适合对跨语言视觉语义理解感兴趣的开发者与研究者
SemEval-2025 Task 1 要求根据给定的名词复合词（可能含习语意义）对图像进行排序，涵盖英语和巴西葡萄牙语。本文利用生成式大语言模型（LLMs）和多语言CLIP模型增强习语复合词的表征。首先由LLMs生成潜在习语的隐喻含义，丰富其语义解释；随后使用多语言CLIP模型对这些含义进行编码，作为图像排序的语义表示。通过对比学习与数据增强技术对嵌入向量进行微调以提升性能。实验表明，该方法提取的多模态表示优于仅基于原始名词复合词的表征。尽管微调策略表现良好，但未微调的嵌入效果更优。相关代码已开源：https://github.com/tongwu17/SemEval-2025-Task1-UoR-NCL。
原文摘要 · Abstract (English)
SemEval-2025 Task 1 focuses on ranking images based on their alignment with a given nominal compound that may carry idiomatic meaning in both English and Brazilian Portuguese. To address this challenge, this work uses generative large language models (LLMs) and multilingual CLIP models to enhance idiomatic compound representations. LLMs generate idiomatic meanings for potentially idiomatic compounds, enriching their semantic interpretation. These meanings are then encoded using multilingual CLIP models, serving as representations for image ranking. Contrastive learning and data augmentation techniques are applied to fine-tune these embeddings for improved performance. Experimental results show that multimodal representations extracted through this method outperformed those based solely on the original nominal compounds. The fine-tuning approach shows promising outcomes but is less effective than using embeddings without fine-tuning. The source code used in this paper is available at https://github.com/tongwu17/SemEval-2025-Task1-UoR-NCL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。