arXiv:2409.17728cs.CVcs.AI2024-09NeurIPS被引 1

提出一种新剪枝方法,解决相机激光雷达融合模型的特征冗余问题。

AlterMOMA: Fusion Redundancy Pruning for Camera-LiDAR Fusion Models with Alternative Modality Masking

  • 通过交替屏蔽模态,检测冗余参数
  • 在nuScene和KITTI上实现最优剪枝性能
  • 适合需要轻量化融合模型的研究者

相机-激光雷达融合模型显著提升了自动驾驶中的感知性能。融合机制利用各模态优势并弥补其不足。实践中,融合模型常采用预训练的单模态主干网络以提高训练效率。然而,我们指出直接加载单模态预训练主干会因融合机制本身引入跨模态特征冗余。现有剪枝方法专为单模态设计,难以有效识别此类冗余参数。为此,本文提出新颖的剪枝框架AlterMOMA,通过在各模态上实施交替屏蔽,识别冗余参数。当某一模态参数被屏蔽时,其主干特征缺失迫使模型重新激活另一主干的冗余特征。由此可识别冗余特征及其对应参数。我们基于损失变化设计重要性评分函数AlterEva,评估参数剪枝价值。在nuScenes和KITTI数据集上,涵盖多种任务、基线模型与剪枝算法的大量实验表明,AlterMOMA优于现有方法,达到当前最优性能。

原文摘要 · Abstract (English)

Camera-LiDAR fusion models significantly enhance perception performance in autonomous driving. The fusion mechanism leverages the strengths of each modality while minimizing their weaknesses. Moreover, in practice, camera-LiDAR fusion models utilize pre-trained backbones for efficient training. However, we argue that directly loading single-modal pre-trained camera and LiDAR backbones into camera-LiDAR fusion models introduces similar feature redundancy across modalities due to the nature of the fusion mechanism. Unfortunately, existing pruning methods are developed explicitly for single-modal models, and thus, they struggle to effectively identify these specific redundant parameters in camera-LiDAR fusion models. In this paper, to address the issue above on camera-LiDAR fusion models, we propose a novelty pruning framework Alternative Modality Masking Pruning (AlterMOMA), which employs alternative masking on each modality and identifies the redundant parameters. Specifically, when one modality parameters are masked (deactivated), the absence of features from the masked backbone compels the model to reactivate previous redundant features of the other modality backbone. Therefore, these redundant features and relevant redundant parameters can be identified via the reactivation process. The redundant parameters can be pruned by our proposed importance score evaluation function, Alternative Evaluation (AlterEva), which is based on the observation of the loss changes when certain modality parameters are activated and deactivated. Extensive experiments on the nuScene and KITTI datasets encompassing diverse tasks, baseline models, and pruning algorithms showcase that AlterMOMA outperforms existing pruning methods, attaining state-of-the-art performance.

多模态融合模型剪枝自动驾驶特征冗余

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。