提出分层补偿压缩方法,实现3D视觉语言模型98%的令牌压缩。
HCC-3D: Hierarchical Compensatory Compression for 98% 3D Token Reduction in Vision-Language Models
- 通过全局结构压缩与自适应细节挖掘,分两步压缩3D令牌。
- 压缩率达98%,仍保持领先性能,效率与精度双提升。
- 适合需要高效3D理解的场景,如机器人感知、自动驾驶。
三维理解近年来受到广泛关注,借助视觉-语言模型(VLMs)实现点云与文本数据间的多模态推理。现有3D-VLMs直接将点云嵌入为3D令牌,沿用具备强大推理能力的大规模2D-VLM框架,但该方法计算开销巨大,我们发现瓶颈在于在大语言模型(LLM)部分处理全部3D令牌。这引发关键问题:如何在保留核心信息的前提下降低3D令牌带来的计算负担?为此,我们提出分层补偿压缩(HCC-3D),以高效压缩3D令牌并保持关键细节。具体而言,首先设计全局查询进行全局结构压缩(GSC),将所有3D令牌压缩为少数关键令牌,保留整体结构信息;随后,为弥补GSC的信息损失,引入自适应细节挖掘(ADM)模块,通过互补评分选择性地重新压缩显著但被忽略的特征。大量实验表明,HCC-3D相比以往3D-VLMs实现了约98%的压缩率,同时达到新的最先进性能,在效率与性能上均有显著提升。
原文摘要 · Abstract (English)
3D understanding has drawn significant attention recently, leveraging Vision-Language Models (VLMs) to enable multi-modal reasoning between point cloud and text data. Current 3D-VLMs directly embed the 3D point clouds into 3D tokens, following large 2D-VLMs with powerful reasoning capabilities. However, this framework has a great computational cost limiting its application, where we identify that the bottleneck lies in processing all 3D tokens in the Large Language Model (LLM) part. This raises the question: how can we reduce the computational overhead introduced by 3D tokens while preserving the integrity of their essential information? To address this question, we introduce Hierarchical Compensatory Compression (HCC-3D) to efficiently compress 3D tokens while maintaining critical detail retention. Specifically, we first propose a global structure compression (GSC), in which we design global queries to compress all 3D tokens into a few key tokens while keeping overall structural information. Then, to compensate for the information loss in GSC, we further propose an adaptive detail mining (ADM) module that selectively recompresses salient but under-attended features through complementary scoring. Extensive experiments demonstrate that HCC-3D not only achieves extreme compression ratios (approximately 98%) compared to previous 3D-VLMs, but also achieves new state-of-the-art performance, showing the great improvements on both efficiency and performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。