用轻量Mamba模型实现多模态感知,缺损数据也能自适应补偿。
GRAM-MAMBA: Holistic Feature Alignment for Wireless Perception with Adaptive Low-Rank Compensation
- 用Mamba处理时序数据,配合成对模态对齐矩阵提升融合效率。
- 缺失模态时仅训练不到0.2%参数,性能提升24.5%。
- 适合资源受限的物联网场景,如智能家居与健康监测。
多模态融合对物联网感知至关重要,广泛应用于智能家居、智能交通、工业自动化和医疗健康。然而现有系统常面临模型复杂度高、单向模态对齐忽略跨模态关系、传感器数据缺失时鲁棒性差等问题,制约了真实物联网环境下的高效可靠感知。为此,我们提出GRAM-MAMBA框架:采用线性复杂度的Mamba模型高效处理传感器时序数据,结合优化的GRAM矩阵策略实现模态间成对对齐,克服传统单模态对齐缺陷;受低秩适配(LoRA)启发,引入自适应低秩层补偿机制,在模型训练后应对缺失模态——冻结预训练核心及无关自适应层,仅微调与可用模态及融合过程相关的部分。大量实验验证其有效性:在SPAWC2021室内定位数据集上,预训练模型误差低于基线,适应缺失模态时性能提升24.5%,仅训练少于0.2%参数;在USC-HAD人体行为识别数据集上,取得93.55% F1与93.81%整体准确率(OA),优于已有方法;更新策略使F1提升23%,且训练参数不足0.3%。结果表明,GRAM-MAMBA在资源受限环境下具备高效稳健的多模态感知潜力。
原文摘要 · Abstract (English)
Multi-modal fusion is crucial for Internet of Things (IoT) perception, widely deployed in smart homes, intelligent transport, industrial automation, and healthcare. However, existing systems often face challenges: high model complexity hinders deployment in resource-constrained environments, unidirectional modal alignment neglects inter-modal relationships, and robustness suffers when sensor data is missing. These issues impede efficient and robust multimodal perception in real-world IoT settings. To overcome these limitations, we propose GRAM-MAMBA. This framework utilizes the linear-complexity Mamba model for efficient sensor time-series processing, combined with an optimized GRAM matrix strategy for pairwise alignment among modalities, addressing the shortcomings of traditional single-modality alignment. Inspired by Low-Rank Adaptation (LoRA), we introduce an adaptive low-rank layer compensation strategy to handle missing modalities post-training. This strategy freezes the pre-trained model core and irrelevant adaptive layers, fine-tuning only those related to available modalities and the fusion process. Extensive experiments validate GRAM-MAMBA's effectiveness. On the SPAWC2021 indoor positioning dataset, the pre-trained model shows lower error than baselines; adapting to missing modalities yields a 24.5% performance boost by training less than 0.2% of parameters. On the USC-HAD human activity recognition dataset, it achieves 93.55% F1 and 93.81% Overall Accuracy (OA), outperforming prior work; the update strategy increases F1 by 23% while training less than 0.3% of parameters. These results highlight GRAM-MAMBA's potential for achieving efficient and robust multimodal perception in resource-constrained environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。