用新模型实现实时非视距视频重建,快且清晰。
TransiT: Transient Transformer for Non-line-of-sight Videography
- 设计瞬态变压器架构,压缩时间维度降计算开销
- 16×16稀疏数据下实现64×64分辨率、10帧/秒实时重建
- 融合合成与真实数据,适合自动驾驶和救援场景
使用非视距(NLOS)成像进行高质量、高速视频采集对自主导航、避障及灾后搜救至关重要。当前方法需在帧率与图像质量间权衡:提高帧率通常通过缩短每点扫描时间或降低扫描密度实现,但会降低单帧信息密度,且快速扫描会降低信噪比,不同系统还存在不同失真特征。本文提出新型瞬态变压器架构TransiT,可在快速扫描条件下实现实时NLOS恢复。TransiT直接压缩输入瞬态的时间维度提取特征,降低计算成本并满足高帧率需求;同时采用特征融合机制与时空Transformer捕捉视频特征;并通过迁移学习弥合合成数据与真实测量数据之间的差距。在真实实验中,TransiT仅需每点0.4毫秒曝光时间、16×16稀疏测量数据,即可重建出64×64分辨率、10帧每秒的NLOS视频。代码与数据集将公开共享。
原文摘要 · Abstract (English)
High quality and high speed videography using Non-Line-of-Sight (NLOS) imaging benefit autonomous navigation, collision prevention, and post-disaster search and rescue tasks. Current solutions have to balance between the frame rate and image quality. High frame rates, for example, can be achieved by reducing either per-point scanning time or scanning density, but at the cost of lowering the information density at individual frames. Fast scanning process further reduces the signal-to-noise ratio and different scanning systems exhibit different distortion characteristics. In this work, we design and employ a new Transient Transformer architecture called TransiT to achieve real-time NLOS recovery under fast scans. TransiT directly compresses the temporal dimension of input transients to extract features, reducing computation costs and meeting high frame rate requirements. It further adopts a feature fusion mechanism as well as employs a spatial-temporal Transformer to help capture features of NLOS transient videos. Moreover, TransiT applies transfer learning to bridge the gap between synthetic and real-measured data. In real experiments, TransiT manages to reconstruct from sparse transients of $16 \times 16$ measured at an exposure time of 0.4 ms per point to NLOS videos at a $64 \times 64$ resolution at 10 frames per second. We will make our code and dataset available to the community.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。