轻量级网络解决事件相机色彩插值难题,适合手机端高效运行。
Lightweight Quad Bayer HybridEVS Demosaicing via State Space Augmented Cross-Attention
- 分两阶段处理事件像素补全与色彩插值,提升任务可解性。
- 在7个数据集上平均优于前代方法,参数减少1.86倍、计算量降3.29倍。
- 结合状态空间模型与位置先验,实现高效全局依赖建模。
事件相机如混合事件视觉传感器(HybridEVS)通过异步事件捕捉亮度变化,适用于移动摄影。但将四色块彩色滤光阵列(Quad Bayer CFA)与无颜色信息的事件像素结合时,会导致去马赛克过程中的混叠与伪影。现有方法难以在资源受限的移动端有效解决此问题。为此,我们提出轻量级双阶段网络TSANet,采用状态空间增强的交叉注意力机制,分别处理事件像素补全与去马赛克。进一步设计轻量级Cross-Swin状态块,利用位置先验优化去马赛克,并通过线性复杂度的状态空间模型增强全局依赖。实验表明,TSANet在模拟与真实HybridEVS数据上均表现优异,于七个不同数据集上平均性能超越当前最优方法DemosaicFormer,在PSNR与SSIM上均有提升,同时参数量降低1.86倍,计算成本减少3.29倍。该方法为移动端高效图像去马赛克提供了新路径。代码见附录。
原文摘要 · Abstract (English)
Event cameras like the Hybrid Event-based Vision Sensor (HybridEVS) camera capture brightness changes as asynchronous "events" instead of frames, offering advanced application on mobile photography. However, challenges arise from combining a Quad Bayer Color Filter Array (CFA) sensor with event pixels lacking color information, resulting in aliasing and artifacts on the demosaicing process before downstream application. Current methods struggle to address these issues, especially on resource-limited mobile devices. In response, we introduce \textbf{TSANet}, a lightweight \textbf{T}wo-stage network via \textbf{S}tate space augmented cross-\textbf{A}ttention, which can handle event pixels inpainting and demosaicing separately, leveraging the benefits of dividing complex tasks into manageable subtasks. Furthermore, we introduce a lightweight Cross-Swin State Block that uniquely utilizes positional prior for demosaicing and enhances global dependencies through the state space model with linear complexity. In summary, TSANet demonstrates excellent demosaicing performance on both simulated and real data of HybridEVS while maintaining a lightweight model, averaging better results than the previous state-of-the-art method DemosaicFormer across seven diverse datasets in both PSNR and SSIM, while respectively reducing parameter and computation costs by $1.86\times$ and $3.29\times$. Our approach presents new possibilities for efficient image demosaicing on mobile devices. Code is available in the supplementary materials.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。