用自回归生成少样本向量，实现高效文档检索
CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embedding
- 自回归生成多向量嵌入，仅需几十个视觉标记
- 相比传统方法减少30-155倍标记数，性能不降
- 适合需要快速检索和低存储的文档系统
尽管多模态大语言模型在视觉文档检索（VDR）中通过生成高质量多向量嵌入展现出巨大潜力，但每页使用数千个视觉标记带来的巨大存储开销限制了其在真实场景中的应用。为此，我们提出一种自回归生成方法 CausalEmbed，通过在对比学习中引入迭代边缘损失，促使嵌入模型学习紧凑且结构化的表示。该方法仅需数十个视觉标记即可实现高效的 VDR 任务，在多种骨干网络和基准上保持高度竞争力，标记数量减少 30–155 倍。理论分析与实证结果表明，自回归嵌入生成在训练效率和推理可扩展性方面具有独特优势。CausalEmbed 提出了一种灵活的测试时缩放策略，为多向量 VDR 表示提供了生成范式新思路。代码已开源：https://github.com/Z1zs/Causal-Embed。
原文摘要 · Abstract (English)
Although Multimodal Large Language Models (MLLMs) have shown remarkable potential in Visual Document Retrieval (VDR) through generating high-quality multi-vector embeddings, the substantial storage overhead caused by representing a page with thousands of visual tokens limits their practicality in real-world applications. To address this challenge, we propose an auto-regressive generation approach, CausalEmbed, for constructing multi-vector embeddings. By incorporating iterative margin loss during contrastive training, CausalEmbed encourages the embedding models to learn compact and well-structured representations. Our method enables efficient VDR tasks using only dozens of visual tokens, achieving a 30-155x reduction in token count while maintaining highly competitive performance across various backbones and benchmarks. Theoretical analysis and empirical results demonstrate the unique advantages of auto-regressive embedding generation in terms of training efficiency and scalability at test time. As a result, CausalEmbed introduces a flexible test-time scaling strategy for multi-vector VDR representations and sheds light on the generative paradigm within multimodal document retrieval. Our code is available at https://github.com/Z1zs/Causal-Embed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。