提出轻量级二值化Mamba-Transformer,高效实现混合事件视觉传感器去马赛克。
Binarized Mamba-Transformer for Lightweight Quad Bayer HybridEVS Demosaicing
- 融合Mamba与Swin Transformer,兼顾全局与局部依赖建模。
- 二值化投影+全精度选择扫描,计算量降低78%以上。
- 适合移动端部署,实测在Edge设备上运行速度快、效果优。
Quad Bayer去马赛克是推动混合事件视觉传感器(HybridEVS)广泛应用的核心挑战。现有基于学习的方法虽通过建模长程依赖取得良好效果,但计算复杂度高,难以在移动设备上部署。为此,本文提出一种轻量级Mamba-based二值神经网络——BMTNet,用于高效且高性能地处理HybridEVS RAW图像的去马赛克任务。首先,设计了一种混合二值化Mamba-Transformer架构(BMTNet),结合Mamba与Swin Transformer的优势,有效捕捉全局与局部依赖。其次,提出二值化Mamba(Bi-Mamba),将所有投影二值化,同时保留核心选择性扫描(Selective Scan)的全精度计算,并引入额外全局视觉信息以增强上下文并缓解精度损失。定量与定性实验表明,BMTNet在性能和计算效率方面均表现优异,为真实场景下的边缘设备提供轻量化解决方案。代码与模型已开源于 https://github.com/Clausy9/BMTNet。
原文摘要 · Abstract (English)
Quad Bayer demosaicing is the central challenge for enabling the widespread application of Hybrid Event-based Vision Sensors (HybridEVS). Although existing learning-based methods that leverage long-range dependency modeling have achieved promising results, their complexity severely limits deployment on mobile devices for real-world applications. To address these limitations, we propose a lightweight Mamba-based binary neural network designed for efficient and high-performing demosaicing of HybridEVS RAW images. First, to effectively capture both global and local dependencies, we introduce a hybrid Binarized Mamba-Transformer architecture that combines the strengths of the Mamba and Swin Transformer architectures. Next, to significantly reduce computational complexity, we propose a binarized Mamba (Bi-Mamba), which binarizes all projections while retaining the core Selective Scan in full precision. Bi-Mamba also incorporates additional global visual information to enhance global context and mitigate precision loss. We conduct quantitative and qualitative experiments to demonstrate the effectiveness of BMTNet in both performance and computational efficiency, providing a lightweight demosaicing solution suited for real-world edge devices. Our codes and models are available at https://github.com/Clausy9/BMTNet.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。