让视觉检索快4倍,不用训练就能压缩海量向量。
Visual RAG Toolkit: Scaling Multi-Vector Visual Retrieval with Training-Free Pooling and Multi-Stage Search
- 用无训练的分块平均法压缩每页图像向量,从千级减到十级。
- 两阶段检索保持高精度,召回率@5/10仅轻微下降,吞吐提升约4倍。
- 适合需要高效视觉检索的开发者,尤其关注实用性和部署成本。
多向量视觉检索器(如ColPali类晚期交互模型)精度高,但因每页生成数千个向量,导致索引与搜索成本急剧上升。本文提出Visual RAG Toolkit,通过无需训练的、模型感知的静态空间池化(包括轻量滑动窗口平均)将图像块嵌入压缩为紧凑的瓦片级与全局表示,用于快速候选生成;随后使用完整多向量嵌入进行精确的MaxSim重排序。该设计将向量间比较次数降至二次方级别,使每页存储向量数由数千降至数十,且无需后训练、适配器或知识蒸馏。在有限的ViDoRe v2基准数据集上,两阶段检索在常见截断值(k ≤ 10)下基本保持NDCG与召回率@5/10,同时显著提升吞吐量(约4倍QPS),仅在极大k值时敏感。工具包还提供鲁棒预处理(高分辨率PDF转图像、可选边距/空白区域裁剪、仅索引视觉标记)、可复现评估流程,并支持二、三、级联等检索变体快速探索。通过聚焦常见截断场景,显著降低硬件门槛,使前沿视觉检索更易落地实践。
原文摘要 · Abstract (English)
Multi-vector visual retrievers (e.g., ColPali-style late interaction models) deliver strong accuracy, but scale poorly because each page yields thousands of vectors, making indexing and search increasingly expensive. We present Visual RAG Toolkit, a practical system for scaling visual multi-vector retrieval with training-free, model-aware pooling and multi-stage retrieval. Motivated by Matryoshka Embeddings, our method performs static spatial pooling - including a lightweight sliding-window averaging variant - over patch embeddings to produce compact tile-level and global representations for fast candidate generation, followed by exact MaxSim reranking using full multi-vector embeddings. Our design yields a quadratic reduction in vector-to-vector comparisons by reducing stored vectors per page from thousands to dozens, notably without requiring post-training, adapters, or distillation. Across experiments with interaction-style models such as ColPali and ColSmol-500M, we observe that over the limited ViDoRe v2 benchmark corpus 2-stage retrieval typically preserves NDCG and Recall @ 5/10 with minimal degradation, while substantially improving throughput (approximately 4x QPS); with sensitivity mainly at very large k. The toolkit additionally provides robust preprocessing - high resolution PDF to image conversion, optional margin/empty-region cropping and token hygiene (indexing only visual tokens) - and a reproducible evaluation pipeline, enabling rapid exploration of two-, three-, and cascaded retrieval variants. By emphasizing efficiency at common cutoffs (e.g., k <= 10), the toolkit lowers hardware barriers and makes state-of-the-art visual retrieval more accessible in practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。