提出连续跨模态融合网络,让颜色与深度信息全程协同学习。
MSNN-LINet: Cross-Modal Learning via Continuous Linear Integration

- 用线性融合卷积实现每层都跨模态整合,避免早期或晚期融合的缺陷。
- 在SUN RGB-D上达到45.2%准确率,预训练后提升至49.6%。
- 适合做多模态视觉任务、对融合机制有研究兴趣的读者。
我们提出LINet(线性整合网络),一种用于RGB-D场景分类的多流神经网络(MSNN)。现有方法将特征融合视为离散、随意的操作:早期融合过早纠缠表示,晚期融合直到最后一层才分离,混合或注意力方法则需猜测中间融合块的位置。LINet通过维持三个独立并行流(RGB、深度、整合)解决这一结构困境,其中新型线性整合卷积(LIConv2d)可在每一层实现连续跨模态学习。整合流接收两个模态流的原始滤波信号,在非线性激活前进行组合,受体细胞整合后决定放电的机制启发。连续融合暴露了一个关键初始化问题:Kaiming初始化使桥接权重梯度混乱,导致梯度流动异常,表现为类似过拟合的失败模式。采用1/N常数初始化可缓解该问题。我们引入渐进式模态丢弃,一种适应连续融合的课程学习策略,空白概率从零逐渐增加,防止路径坍塌——一种负向协同学习现象——迫使各流建立稳健的独立表示。在未预训练的SUN RGB-D 19类场景分类任务中,从头训练的LINet达到45.2%平均类别准确率,优于此前从头训练结果;在域内RGB-D数据集ScanNet上预训练后进一步提升至49.6%。
原文摘要 · Abstract (English)
We present LINet (Linear Integration Network), a Multi-Stream Neural Network (MSNN) for RGB-D scene classification. Current multi-modal architectures treat feature fusion as a discrete, ad-hoc event: early fusion entangles representations prematurely, late fusion isolates them until the final layer, and hybrid or attention-based methods require architectural guesswork to place intermediate fusion blocks. LINet addresses this structural compromise by maintaining three dedicated parallel streams (RGB, depth, and integration) where a novel Linear Integration Convolution (LIConv2d) operator enables continuous cross-modal learning at every layer. The integration stream receives raw filtered signals from both modality streams and combines them before the nonlinear activation threshold, conceptually inspired by somatic integration preceding the neuronal firing decision. Implementing continuous integration exposes a critical initialization pathology: Kaiming initialization of the bridging weights scrambles gradients before they reach the stream backbones, producing a failure mode that resembles overfitting but is corrupted gradient flow. A 1/N constant initialization mitigates this. We employ progressive modality dropout, a curriculum adapted to continuous fusion in which blanking probability increases from zero, preventing pathway collapse, a form of negative co-learning, by forcing robust independent stream representations. Trained from scratch on SUN RGB-D 19-class scene classification, LINet reaches 45.2% mean class accuracy at ResNet18 scale, outperforming prior from-scratch results, and rises to 49.6% with in-domain RGB-D (ScanNet) pretraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。