用双向注意力融合高分辨彩色与低分辨遥感光谱图，提升材料识别准确率。
Bidirectional Cross-Attention Fusion of High-Resolution RGB and Low-Resolution Hyperspectral Inputs for Multimodal Semantic Segmentation
- 通过原生网格双向注意力对齐彩色与光谱数据，避免上采样损失
- 在SpectralWaste数据集上达75.4%精度，速度55帧/秒
- 适用于工业级实时材料分类，代码已开源
多模态语义分割需融合空间分辨率与通道维度差异的异构传感器信息。高分辨率彩色图像（RGB）提供精细空间结构，但难以区分视觉相似材料；高光谱成像（HSI）具备强区分性光谱特征，但空间分辨率较低。本文提出双向交叉注意力融合（BCAF），通过局部化双向交叉注意力在各自原生网格上对齐RGB与低分辨率HSI，避免预上采样或早期光谱信息坍缩。BCAF采用两个独立主干网络：标准Swin Transformer处理RGB，HSI适配的Swin主干通过3D令牌化和光谱自注意力保持光谱结构。尽管评估聚焦于RGB-HSI融合，但该方法具有模态无关性，可推广至其他共注册的高通道辅助传感器。在基准数据集SpectralWaste上，BCAF实现75.4%的分割精度，速度达55帧/秒。进一步在新型工业数据集K3I-Cycling（已部分发布于Fordatis）上测试，材料分类达62.3% mIoU，塑料类型分类达66.2% mIoU。结果表明，在实时约束下保留原生空间细节与光谱结构可显著提升多模态分割性能。代码与模型检查点已在https://github.com/jonasvilhofunk/BCAF_2026公开。
原文摘要 · Abstract (English)
Multimodal semantic segmentation with heterogeneous sensors must reconcile complementary information across modalities that differ in spatial resolution and channel dimensionality. In particular, high-resolution RGB imaging provides detailed spatial structure but often fails to distinguish visually similar materials, whereas hyperspectral imaging (HSI) provides discriminative spectral signatures but at lower spatial resolution. We present Bidirectional Cross-Attention Fusion (BCAF), which aligns high-resolution RGB with low-resolution HSI at their native grids via localized, bidirectional cross-attention, avoiding pre-upsampling or early spectral collapse. BCAF uses two independent backbones: a standard Swin Transformer for RGB and an HSI-adapted Swin backbone that preserves spectral structure through 3D tokenization with spectral self-attention. Although our evaluation targets RGB-HSI fusion, BCAF is modality-agnostic and applies to co-registered RGB with lower-resolution, high-channel auxiliary sensors. On the benchmark SpectralWaste dataset, BCAF delivers strong performance, achieving 75.4% at 55 images/s. We further evaluate a novel industrial dataset: K3I-Cycling (first RGB subset already released on Fordatis). On this dataset, BCAF reaches 62.3% mIoU for material segmentation (paper, metal, plastic, etc.) and 66.2% mIoU for plastic-type segmentation (PET, PP, HDPE, LDPE, PS, etc.). These results show that preserving native-grid spatial detail and spectral structure improves multimodal segmentation under real-time constraints. Code and model checkpoints are publicly available at https://github.com/jonasvilhofunk/BCAF_2026.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。