提出Fast3D框架，用轻量预测加速3D多模态大模型推理。
Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding
- 通过全局注意力预测和自适应采样，动态剪枝冗余3D视觉标记。
- 在5个基准上验证，高剪枝率下仍保持优异性能。
- 无需修改原模型，适合部署在资源受限的3D场景理解任务中。
尽管3D多模态大语言模型（MLLMs）展现出强大的场景理解能力，其实际部署因计算效率低下而受阻。主要瓶颈在于为全面表示3D场景所需的大量对象中心视觉标记。虽然视觉标记剪枝在2D MLLMs中已显成效，但因其与3D标记结构存在本质差异，该方法在3D领域尚未被探索。本文揭示两个关键洞察：(1) 3D对象级标记表示存在显著冗余，类似于2D中的块级冗余；(2) 全局注意力模式对识别3D上下文中的非必要标记具有强预测力。基于此，我们提出Fast3D——一个针对3D MLLMs的即插即用视觉标记剪枝框架，包含两项技术创新：(1) 全局注意力预测（GAP），利用轻量神经网络学习目标模型的全局注意力分布，实现高效标记重要性估计；(2) 样本自适应视觉标记剪枝（SAP），通过基于注意力的复杂度评估引入动态标记预算，自动调整各层剪枝比例。两项技术均不修改目标模型参数。在五个基准上的广泛评估验证了Fast3D的有效性，尤其在高视觉标记剪枝率下表现突出。代码开源于https://github.com/wencan25/Fast3D。
原文摘要 · Abstract (English)
While 3D Multi-modal Large Language Models (MLLMs) demonstrate remarkable scene understanding capabilities, their practical deployment faces critical challenges due to computational inefficiency. The key bottleneck stems from processing excessive object-centric visual tokens required for comprehensive 3D scene representation. Although visual token pruning has shown promise in accelerating 2D MLLMs, its applicability to 3D domains remains largely unexplored due to fundamental disparities in token structures. In this paper, we reveal two critical insights: (1) Significant redundancy exists in object-level 3D token representations, analogous to patch-level redundancy in 2D systems; (2) Global attention patterns exhibit strong predictive power for identifying non-essential tokens in 3D contexts. Building on these observations, we propose Fast3D, a plug-and-play visual token pruning framework for 3D MLLMs featuring two technical innovations: (1) Global Attention Prediction (GAP), where a lightweight neural network learns to predict the global attention distributions of the target model, enabling efficient token importance estimation for precise pruning guidance; (2) Sample-Adaptive visual token Pruning (SAP), which introduces dynamic token budgets through attention-based complexity assessment, automatically adjusting layer-wise pruning ratios based on input characteristics. Both of these two techniques operate without modifying the parameters of the target model. Extensive evaluations across five benchmarks validate the effectiveness of Fast3D, particularly under high visual token pruning ratios. Code is available at https://github.com/wencan25/Fast3D
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。