将复杂图文生成拆解为子维度，精准匹配查询细节。
Cross-modal RAG: Sub-dimensional Text-to-Image Retrieval-Augmented Generation
- 将查询与图像分解为子维度，实现细粒度匹配
- 混合稀疏与稠密检索，找到互补图像组合
- 适合需要精准控制生成内容的研究者
文本到图像生成日益需要访问领域特定、细粒度且快速演化的知识，而预训练模型难以完全覆盖，因此需引入检索机制。现有检索增强生成（RAG）方法通过检索全局相关图像来应对，但在复杂查询中，单张图像无法包含所有所需元素时表现不佳。本文提出跨模态RAG框架，将查询与图像分解为子维度成分，实现子查询感知的检索与生成。方法采用混合检索策略——结合子维度稀疏检索器与稠密检索器，以识别一组帕累托最优图像，每张图像贡献查询中的互补部分。生成阶段，多模态大语言模型被引导选择性地基于与特定子查询对齐的视觉特征进行条件生成，确保子查询感知的图像合成。在MS-COCO、Flickr30K、WikiArt、CUB和ImageNet-LT上的大量实验表明，跨模态RAG显著优于现有基线，在检索性能上提升明显，并进一步提升生成质量，同时保持高效。
原文摘要 · Abstract (English)
Text-to-image generation increasingly demands access to domain-specific, fine-grained, and rapidly evolving knowledge that pretrained models cannot fully capture, necessitating the integration of retrieval methods. Existing Retrieval-Augmented Generation (RAG) methods attempt to address this by retrieving globally relevant images, but they fail when no single image contains all desired elements from a complex user query. We propose Cross-modal RAG, a novel framework that decomposes both queries and images into sub-dimensional components, enabling subquery-aware retrieval and generation. Our method introduces a hybrid retrieval strategy - combining a sub-dimensional sparse retriever with a dense retriever - to identify a Pareto-optimal set of images, each contributing complementary aspects of the query. During generation, a multimodal large language model is guided to selectively condition on relevant visual features aligned to specific subqueries, ensuring subquery-aware image synthesis. Extensive experiments on MS-COCO, Flickr30K, WikiArt, CUB, and ImageNet-LT demonstrate that Cross-modal RAG significantly outperforms existing baselines in the retrieval and further contributes to generation quality, while maintaining high efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。