提出零样本剪枝与稀疏注意力,让边缘设备多模态推理更省电高效。
Modality-Aware Zero-Shot Pruning and Sparse Attention for Efficient Multimodal Edge Inference
- 训练时学习模态感知重要性,部署时无需微调直接剪枝。
- 在三种模型上减少15%计算量,模态缺失下准确率提升最多18%。
- 适合资源受限的多传感器边缘设备,尤其应对电源波动和传感器失效。
边缘设备越来越多地运行多模态感知流水线,但需在波动的功耗预算和不可预测的传感器失效条件下保持准确性。现有剪枝方法在此场景下表现不佳:通常需要压缩后微调,消耗超过10倍于部署阶段的能耗,且分配的权重重要性是静态的,不考虑实际可用传感器。本文提出SentryFuse框架,通过两个核心组件协同解决上述问题。首先,SentryGate在训练中通过一阶显著性监督学习模态条件下的重要性得分,并在部署时无须微调即可剪枝注意力头和前馈通道。其次,SentryAttend将当前多模态架构中的密集自注意力替换为稀疏分组查询注意力,在三种不同多模态架构上实现总体15%的GFLOPs减少。在三个应用场景和多个多模态主干网络上,SentryGate相比最强剪枝基线平均提升12.7%准确率,模态丢失条件下最高提升18%。SentryFuse共同实现内存减少28.2%,延迟降低最高达1.63倍,且无需额外微调,确立了模态感知零样本压缩在异构边缘硬件上实现多模态智能的可行性。
原文摘要 · Abstract (English)
Edge devices increasingly run multimodal sensing pipelines that must remain accurate despite fluctuating power budgets and unpredictable sensor dropout. Existing pruning methods fail under these conditions: they generally require fine-tuning after compression, consuming over $10\times$ the deployment energy, and they assign static importance scores that are blind to which sensors are present. We present the SentryFuse framework, which addresses both challenges jointly through two key components. First, SentryGate learns modality-conditioned importance scores during training via first-order saliency supervision and then prunes attention heads and feed-forward channels at deployment without fine-tuning. Second, SentryAttend replaces dense self-attention, a key bottleneck in contemporary multimodal architectures, with sparse grouped-query attention, yielding a net 15% reduction in GFLOPs across three different multimodal architectures. Across three applications and multimodal backbones, SentryGate achieves a 12.7% average accuracy improvement over the strongest pruning baseline, and upto to 18% under modality dropout conditions. Together, SentryFuse reduces memory by 28.2% and lowers latency by up to $1.63\times$ without further fine-tuning, establishing modality-aware zero-shot compression as a practical path to multimodal intelligence on heterogeneous edge hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。