用概念漂移和层归一化调优,高效识别网络迷因中的隐喻
Concept Drift Guided LayerNorm Tuning for Efficient Multimodal Metaphor Identification
- 通过跨模态嵌入的球面插值生成新概念,缩小字面与隐喻的差距
- 在MET-Meme数据集上达到最新性能,训练成本远低于生成式方法
- 适合研究多模态理解、隐喻识别及轻量化模型优化的学者
隐喻想象是人类认知与沟通的核心能力。尽管语言隐喻理解已取得进展,但互联网迷因等多模态隐喻因其非传统表达和隐含意义而更具挑战性。现有方法难以弥合字面与隐喻解释之间的鸿沟。基于大语言模型或文生图模型的生成式方法虽具潜力,但计算开销高。本文提出概念漂移引导的层归一化调优(CDGLT),包含两项创新:(1) 概念漂移机制,利用CLIP编码器的跨模态嵌入进行球面线性插值(SLERP),生成偏离原空间的新概念嵌入,缓解字面特征与隐喻任务间的差异;(2) 自适应提示构建策略,改进预训练语言模型在多模态隐喻识别中的特征提取与融合方式。CDGLT在MET-Meme基准上达到最优性能,同时显著降低训练成本。消融实验证明了概念漂移与自适应层归一化调优的有效性。代码已开源。
原文摘要 · Abstract (English)
Metaphorical imagination, the ability to connect seemingly unrelated concepts, is fundamental to human cognition and communication. While understanding linguistic metaphors has advanced significantly, grasping multimodal metaphors, such as those found in internet memes, presents unique challenges due to their unconventional expressions and implied meanings. Existing methods for multimodal metaphor identification often struggle to bridge the gap between literal and figurative interpretations. Additionally, generative approaches that utilize large language models or text-to-image models, while promising, suffer from high computational costs. This paper introduces \textbf{C}oncept \textbf{D}rift \textbf{G}uided \textbf{L}ayerNorm \textbf{T}uning (\textbf{CDGLT}), a novel and training-efficient framework for multimodal metaphor identification. CDGLT incorporates two key innovations: (1) Concept Drift, a mechanism that leverages Spherical Linear Interpolation (SLERP) of cross-modal embeddings from a CLIP encoder to generate a new, divergent concept embedding. This drifted concept helps to alleviate the gap between literal features and the figurative task. (2) A prompt construction strategy, that adapts the method of feature extraction and fusion using pre-trained language models for the multimodal metaphor identification task. CDGLT achieves state-of-the-art performance on the MET-Meme benchmark while significantly reducing training costs compared to existing generative methods. Ablation studies demonstrate the effectiveness of both Concept Drift and our adapted LN Tuning approach. Our method represents a significant step towards efficient and accurate multimodal metaphor understanding. The code is available: \href{https://github.com/Qianvenh/CDGLT}{https://github.com/Qianvenh/CDGLT}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。