用手机多摄像头加滤光片,提升光谱成像精度30%。
Modulate and Reconstruct: Learning Hyperspectral Imaging from Misaligned Smartphone Views
- 利用三摄手机搭配特定滤光片,采集多视角光谱数据。
- 相比单摄像头,光谱估计准确率提升30%,顶尖方法再增5%。
- 提出首个多图像光谱重建数据集和轻量对齐模块,适合移动设备应用。
从RGB图像重建高光谱图像在精准色彩还原和材料颜色测量方面具有广阔前景。现有方法多依赖单张RGB图像,限制了重建精度;而现代智能手机普遍配备双摄或三摄系统。本文提出一种新型多图像到高光谱重建(MI-HSR)框架,基于三摄手机系统,其中两颗镜头配备精心设计的光谱滤光片。该配置通过理论与实证分析验证,可获取比传统单相机更完整、更丰富的光谱数据。为支持此新范式,我们构建了首个MI-HSR数据集Doomer,包含三个手机摄像头的对齐图像及一个高光谱参考相机在多种场景下的数据。同时引入轻量级对齐模块,有效融合多视角输入,缓解视差与遮挡引起的伪影。实验表明,该模块可持续提升当前主流高光谱重建方法的性能。总体而言,利用消费级硬件对多视角进行光谱过滤,可实现30%更高的光谱估计精度,且对齐模块使当前最优方法质量再提升5%。结果表明,多视角光谱滤波为更精确、实用的高光谱成像提供了可能。
原文摘要 · Abstract (English)
Hyperspectral reconstruction (HSR) from RGB images is a highly promising direction for accurate color reproduction and material color measurement. While most existing approaches rely on a single RGB image - thereby limiting reconstruction accuracy - the majority of modern smartphones are equipped with two or more cameras. In this work, we propose a novel multi-image-to-hyperspectral reconstruction (MI-HSR) framework that leverages a triple-camera smartphone system, where two lenses are equipped with carefully selected spectral filters. Our easy-to-implement configuration, based on theoretical and empirical analysis, allows to obtain more complete and diverse spectral data than traditional single-chamber setups. To support this new paradigm, we introduce Doomer, the first dataset for MI-HSR, comprising aligned images from three smartphone cameras and a hyperspectral reference camera across diverse scenes. We further introduce a lightweight alignment module for MI-HSR that effectively fuses multi-view inputs while mitigating parallax- and occlusion-induced artifacts. Proposed module demonstrate consistent quality improvements for modern HSR methods. In a nutshell, our setup allows 30% more accurate estimations of spectra compared to an ordinary RGB camera, while the proposed alignment module boosts the reconstruction quality of SotA methods by an additional 5%. Our findings suggest that spectral filtering of multiple views with commodity hardware unlocks more accurate and practical hyperspectral imaging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。