用事件相机+关键帧重建视频,省电又保画质。
IE2Video: Adapting Pretrained Diffusion Models for Event-Based Video Reconstruction
- 结合事件流与稀疏关键帧,用扩散模型生成完整视频
- 相比自回归方法,感知质量提升33%(LPIPS 0.422)
- 跨数据集泛化强,适合低功耗视觉系统部署
监控、机器人和可穿戴设备中的连续视频采集面临能耗瓶颈:传统RGB相机以固定速率采样,功耗高。事件相机虽能以极低功耗实现运动驱动的稀疏感知,但输出为异步事件流而非标准视频。本文提出一种混合采集范式:同时记录稀疏的RGB关键帧与连续事件流,离线重建完整RGB视频,在降低采集功耗的同时保留下游应用所需的视频格式。提出「图像与事件到视频」(IE2Video)任务,即从初始帧和后续事件数据重建视频序列。研究两种架构策略:将自回归模型HyperE2VID用于RGB生成,或通过可学习编码器与低秩适配(LoRA)将事件表示注入预训练文本到视频扩散模型(LTX)。实验表明,扩散模型方法在感知质量上比自回归基线提升33%(LPIPS 0.422 vs 0.283),并在三个事件相机数据集(BS-ERGB、HS-ERGB far/close)上,于32-128帧不同序列长度下均表现出强跨数据集泛化能力,对未见采集配置亦有优异表现。
原文摘要 · Abstract (English)
Continuous video monitoring in surveillance, robotics, and wearable systems faces a fundamental power constraint: conventional RGB cameras consume substantial energy through fixed-rate capture. Event cameras offer sparse, motion-driven sensing with low power consumption, but produce asynchronous event streams rather than RGB video. We propose a hybrid capture paradigm that records sparse RGB keyframes alongside continuous event streams, then reconstructs full RGB video offline -- reducing capture power consumption while maintaining standard video output for downstream applications. We introduce the Image and Event to Video (IE2Video) task: reconstructing RGB video sequences from a single initial frame and subsequent event camera data. We investigate two architectural strategies: adapting an autoregressive model (HyperE2VID) for RGB generation, and injecting event representations into a pretrained text-to-video diffusion model (LTX) via learned encoders and low-rank adaptation. Our experiments demonstrate that the diffusion-based approach achieves 33\% better perceptual quality than the autoregressive baseline (0.283 vs 0.422 LPIPS). We validate our approach across three event camera datasets (BS-ERGB, HS-ERGB far/close) at varying sequence lengths (32-128 frames), demonstrating robust cross-dataset generalization with strong performance on unseen capture configurations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。