arXiv:2604.17054cs.CVcs.AI2026-04中稿 · ance to WACV 2026,…被引 5

无需训练,用提示词控制多模态嵌入,实现图文与矢量图精准检索。

mEOL: Training-Free Instruction-Guided Multimodal Embedder for Vector Graphics and Image Retrieval

论文配图:mEOL: Training-Free Instruction-Guided Multimodal Embedder for Vector Graphics and Image Retrieval
图 1 · 摘自论文原文
  • 通过指令引导大模型将多模态输入压缩为单个词嵌入,实现无训练嵌入。
  • 在文本到矢量图检索任务中超越基于编码器的基线方法,准确率提升12.3%。
  • 适合需要快速部署、关注结构信息的多模态检索场景。

可扩展矢量图形(SVG)既是视觉图像,也包含丰富的几何与布局结构信息,但现有方法常将其光栅化并丢弃符号结构。同时,当前句子嵌入方法虽生成强文本表征,却难以自然拓展至视觉或结构化模态。本文提出一种无需训练、由指令引导的多模态嵌入框架,利用多模态大语言模型(MLLM)将文本、位图图像和SVG代码映射至对齐的嵌入空间。通过特定模态指令与结构化SVG提示控制嵌入方向,无需学习投影头或对比学习训练。核心包含两部分:(1) 多模态显式单字限制(mEOL),指导MLLM将任意多模态输入总结为一个词,其隐藏状态作为紧凑语义嵌入;(2) 语义SVG重写模块,通过渲染图像进行视觉推理,为元素分配有意义标识并简化嵌套结构,揭示原始代码中隐藏的几何与关系线索。基于重构的VGBench构建首个文本到SVG检索基准,实验表明,该方法在零训练条件下优于基于编码器及需训练的多模态基线。结果表明,提示级控制是实现结构感知多模态检索的有效替代方案。

原文摘要 · Abstract (English)

Scalable Vector Graphics (SVGs) function both as visual images and as structured code that encode rich geometric and layout information, yet most methods rasterize them and discard this symbolic organization. At the same time, recent sentence embedding methods produce strong text representations but do not naturally extend to visual or structured modalities. We propose a training-free, instruction-guided multimodal embedding framework that uses a Multimodal Large Language Model (MLLM) to map text, raster images, and SVG code into an aligned embedding space. We control the direction of embeddings through modality-specific instructions and structural SVG cues, eliminating the need for learned projection heads or contrastive training. Our method has two key components: (1) Multimodal Explicit One-word Limitation (mEOL), which instructs the MLLM to summarize any multimodal input into a single token whose hidden state serves as a compact semantic embedding. (2) A semantic SVG rewriting module that assigns meaningful identifiers and simplifies nested SVG elements through visual reasoning over the rendered image, exposing geometric and relational cues hidden in raw code. Using a repurposed VGBench, we build the first text-to-SVG retrieval benchmark and show that our training-free embeddings outperform encoder-based and training-based multimodal baselines. These results highlight prompt-level control as an effective alternative to parameter-level training for structure-aware multimodal retrieval. Project page: https://scene-the-ella.github.io/meol/

多模态嵌入矢量图检索提示工程MLLM

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。