压缩文档图像嵌入，大幅降内存不丢性能
Towards Storage-Efficient Visual Document Retrieval: An Empirical Study on Reducing Patch-Level Embeddings
- 用合并策略替代删除，更适配文档检索需求
- 仅用11.8%内存达原模型98.2%效果
- 适合资源受限场景的高效文档检索
尽管ColPali/ColQwen2在视觉文档检索（VDR）中表现优异，但其将每页文档编码为多个块级嵌入，导致内存占用过高。本实证研究探索在最小性能损失下减少每页嵌入数量的方法。评估了两种标记缩减策略：标记剪枝与标记合并。发现随机剪枝反而优于复杂剪枝方法，但整体效果仍不理想；进一步分析表明，剪枝因缺乏查询信息而不适用于VDR。转向更合适的标记合并策略，通过在三个维度上搜索最优组合，提出Light-ColPali/ColQwen2。该模型仅需原模型11.8%内存即可保持98.2%的检索性能，甚至在2.8%内存下仍保留94.6%的有效性。研究成果为高效VDR提供了重要基准。
原文摘要 · Abstract (English)
Despite the strong performance of ColPali/ColQwen2 in Visualized Document Retrieval (VDR), it encodes each page into multiple patch-level embeddings and leads to excessive memory usage. This empirical study investigates methods to reduce patch embeddings per page at minimum performance degradation. We evaluate two token-reduction strategies: token pruning and token merging. Regarding token pruning, we surprisingly observe that a simple random strategy outperforms other sophisticated pruning methods, though still far from satisfactory. Further analysis reveals that pruning is inherently unsuitable for VDR as it requires removing certain page embeddings without query-specific information. Turning to token merging (more suitable for VDR), we search for the optimal combinations of merging strategy across three dimensions and develop Light-ColPali/ColQwen2. It maintains 98.2% of retrieval performance with only 11.8% of original memory usage, and preserves 94.6% effectiveness at 2.8% memory footprint. We expect our empirical findings and resulting Light-ColPali/ColQwen2 offer valuable insights and establish a competitive baseline for future research towards efficient VDR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。