让多模态嵌入变稀疏，检索速度提升25倍且不丢精度。
PUMA: Post-Hoc Sparsification of Universal Multimodal Embeddings for Efficient Retrieval
- 用稀疏自编码器将稠密嵌入转为紧凑稀疏码，不需重训练主干网络。
- 在五个数据集上，稀疏版检索效果与稠密版相当或更好。
- 适合追求高效多模态检索的工程落地场景。
通用多模态嵌入模型支持跨文本、图像及组合查询的检索，但其稠密表示带来高昂的内存和推理开销。后处理稀疏化可降低这些成本，但在多模态检索中仍研究不足。本文提出PUMA，一种无需重训练主干网络的稀疏自编码器方法：先通过预训练阶段保持稠密点积几何结构，再对稀疏编码器进行检索微调。在涵盖文本到图像和组合图像检索的五个基准上评估，PUMA在Qwen3-VL-Embedding-2B模型上，在四个数据集上的表现与稠密检索无统计差异或更优。我们识别出后处理稀疏化的两类失败模式：预TopK支持不足和检索不匹配的活跃支持。PUMA将向量存储减少8–16倍（FP32），在更大候选池下比精确稠密打分快达25倍，实现高效多模态检索。
原文摘要 · Abstract (English)
Universal multimodal embedders enable retrieval across text, image, and combined queries, but their dense representations incur high memory and inference costs. Post-hoc sparsification could reduce these costs but remains underexplored for multimodal retrieval. We introduce PUMA, a sparse autoencoder recipe that maps universal multimodal embeddings to compact sparse codes without retraining the backbone: a pretraining stage preserves dense dot-product geometry, after which the sparse encoder is fine-tuned for retrieval. We evaluate on five benchmarks covering text-to-image and composed image retrieval. On Qwen3-VL-Embedding-2B, PUMA is statistically indistinguishable from or improves over dense retrieval on four of five datasets. We further identify two failure modes of post-hoc sparsification: insufficient pre-TopK support and retrieval-misaligned active support. PUMA reduces vector storage by 8-16x (FP32) and is up to 25x faster than exact dense scoring on larger candidate pools, enabling efficient multimodal retrieval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。