用生成模型合成数据,让图像检索更懂复杂问题。
Needle: A Generative AI-Powered Multi-modal Database for Answering Complex Natural Language Queries
- 用大模型生成语义匹配的合成数据,统一文本与图像空间
- 在多个基准上超越现有最先进方法,显著提升复杂查询准确率
- 开源可部署,支持任意大模型接入,适合研究与工程落地
多模态数据集(如图像)常缺乏详细描述,难以应对复杂自然语言查询。传统最近邻搜索依赖单一度量空间,但查询与数据嵌入在不同空间,使常规方法失效。现有方法通过联合训练图文向量缓解此问题,但在复杂查询上表现不佳。本文提出基于生成式蒙特卡洛的方法,利用基础模型生成捕捉查询复杂性的合成样本,并将其映射到与多模态数据一致的度量空间。据此构建了名为Needle的图像检索数据库,不依赖对比学习或元数据搜索,而是以合成数据为核心。系统开源且可直接部署,实验表明其在多个基准数据集上显著优于现有最先进文本到图像检索方法。任何基础模型与编码器均可轻松集成至Needle,借力技术进步持续提升性能。
原文摘要 · Abstract (English)
Multi-modal datasets, like those involving images, often miss the detailed descriptions that properly capture the rich information encoded in each item. This makes answering complex natural language queries a major challenge in this domain. In particular, unlike the traditional nearest neighbor search, where the tuples and the query are represented as points in a single metric space, these settings involve queries and tuples embedded in fundamentally different spaces, making the traditional query answering methods inapplicable. Existing literature addresses this challenge for image datasets through vector representations jointly trained on natural language and images. This technique, however, underperforms for complex queries due to various reasons. This paper takes a step towards addressing this challenge by introducing a Generative-based Monte Carlo method that utilizes foundation models to generate synthetic samples that capture the complexity of the natural language query and represent it in the same metric space as the multi-modal data. Following this method, we propose Needle, a database for image data retrieval. Instead of relying on contrastive learning or metadata-searching approaches, our system is based on synthetic data generation to capture the complexities of natural language queries. Our system is open-source and ready for deployment, designed to be easily adopted by researchers and developers. The comprehensive experiments on various benchmark datasets verify that this system significantly outperforms state-of-the-art text-to-image retrieval methods in the literature. Any foundation model and embedder can be easily integrated into Needle to improve the performance, piggybacking on the advancements in these technologies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。