用大模型自动生成语义清晰的图像标识符，提升跨模态检索准确率。
MLLM-Driven Semantic Identifier Generation for Generative Cross-Modal Retrieval
- 让多模态大模型从图文对生成结构化语义标识符。
- 标识符由物体、动作等概念级词元构成，无需扩展词表。
- 添加解释生成作为监督信号，减少幻觉并增强语义对齐。
生成式跨模态检索将检索任务转化为生成任务，近年来随着多模态大语言模型（MLLMs）的发展成为新方向。在该设定下，模型根据文本查询生成对应目标图像的标识符。然而，现有方法通常依赖人工设计字符串ID、基于聚类的标签或需词汇扩展的原子标识符，均面临语义对齐困难或可扩展性差的问题。为此，我们提出一种词汇高效的标识符生成框架，通过提示MLLM从图像-标题对生成结构化语义标识符。这些标识符由对象、动作等概念级词元组成，天然契合模型生成空间，无需修改分词器。此外，我们引入推理引导的监督策略，要求模型为每个标识符生成一句话解释，作为辅助监督信号，有效提升语义定位能力并减少训练中的幻觉现象。
原文摘要 · Abstract (English)
Generative cross-modal retrieval, which treats retrieval as a generation task, has emerged as a promising direction with the rise of Multimodal Large Language Models (MLLMs). In this setting, the model responds to a text query by generating an identifier corresponding to the target image. However, existing methods typically rely on manually crafted string IDs, clustering-based labels, or atomic identifiers requiring vocabulary expansion, all of which face challenges in semantic alignment or scalability.To address these limitations, we propose a vocabulary-efficient identifier generation framework that prompts MLLMs to generate Structured Semantic Identifiers from image-caption pairs. These identifiers are composed of concept-level tokens such as objects and actions, naturally aligning with the model's generation space without modifying the tokenizer. Additionally, we introduce a Rationale-Guided Supervision Strategy, prompting the model to produce a one-sentence explanation alongside each identifier serves as an auxiliary supervision signal that improves semantic grounding and reduces hallucinations during training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。