通过冻结主干模型，实现多模态嵌入的高效几何保持。
jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers
- 仅训练连接模块（占总参数0.35%），冻结文本与非文本编码器
- 支持文本、图像、音频、视频统一嵌入，性能接近顶级大模型
- 适合需要快速部署多模态应用且资源有限的场景
本文提出GELATO（Geometry-preserving Embeddings via Locked Aligned TOwers），一种新型多模态嵌入方法。基于视觉语言模型架构，将非文本编码器适配为向语言模型提供输入，由语言模型生成所有模态的统一嵌入。我们构建了jina-embeddings-v5-omni系列模型，可将文本、图像、音频、视频编码至同一语义空间。该方法在两个Jina Embeddings v5文本模型基础上，新增图像与音频编码器，且保持主干模型完全冻结。仅训练连接组件，占联合模型总参数的0.35%，训练效率远高于全参数微调。语言模型对文本输入的输出与原v5文本模型完全一致。评估表明，GELATO性能接近当前最优水平，与更大规模的多模态嵌入模型表现相当。
原文摘要 · Abstract (English)
In this work, we introduce GELATO (Geometry-preserving Embeddings via Locked Aligned TOwers), a novel approach to multimodal embedding models. We build on the VLM-style architecture, in which non-text encoders are adapted to produce input for a language model, which in turn generates embeddings for all varieties of input. We present the result: the jina-embeddings-v5-omni suite, a pair of models that encode text, image, audio, and video input into a single semantic embedding space. GELATO extends the two Jina Embeddings v5 Text models to support additional modality by adding encoders for images and audio. The backbone text embedding models and the added non-text modality encoders remain frozen. We only trained the connecting components, representing 0.35% of the total weights of the joint model. Training is therefore much more efficient than full-parameter retraining. Additionally, the language model remains effectively unaltered, producing exactly the same embeddings for text inputs as the Jina Embeddings v5 Text models. Our evaluations show that GELATO produces results that are competitive with the state-of-the-art, yielding nearly equal performance to larger multimodal embedding models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。