用事件相机辅助视频压缩,提升运动估计精度。
ENCORE: Event-Assisted Complementary Motion Refinement for Learned Video Compression

- 融合事件相机与RGB帧,分解出共性与模态特异性运动特征。
- 在BS-ERGB数据集上实现20.80%的PSNR提升和22.14%的码率节省。
- 适合需要高精度运动建模的视频压缩场景,尤其在快速运动时表现优异。
学习型视频压缩依赖精准的时序建模以消除相邻帧间的冗余。然而,现有编解码器仅基于离散采样的RGB帧推断运动,易受快速运动、模糊、遮挡、纹理弱、光照低及亮度突变影响。事件相机异步捕捉RGB时间戳之间的细粒度强度变化,可提供帧间动态的补充信息。本文提出ENCORE框架,通过互补运动表示(CMR)将对齐的RGB-事件特征分解为共性和模态特异性运动表征;空间能级与冗余感知校准(SERIC)识别对RGB有新增贡献的事件响应,抑制弱或冗余信号,并预测候选光流修正;能量感知路由(EAR)决定修正作用的位置与强度。事件仅作为运动建模的辅助模态,而RGB仍是唯一编码与重建目标。在BS-ERGB、HQ-EVFI和CED数据集上的实验表明,无论分组长度如何,性能均有稳定提升。在BS-ERGB上,最大实现20.80%的PSNR-RGB增益与22.14%的MS-SSIM-RGB BD-rate节省,其余两数据集也保持显著优势。
原文摘要 · Abstract (English)
Learned video compression relies on accurate temporal modeling to remove redundancy between adjacent frames. However, most existing codecs infer motion solely from discretely sampled RGB frames, making their estimates vulnerable to fast motion, blur, occlusion, weak texture, low illumination, and abrupt brightness changes. Event cameras asynchronously capture fine-grained intensity changes between RGB timestamps and therefore provide complementary evidence about inter-frame dynamics. We propose ENCORE, an Event-Assisted Complementary Motion Refinement framework for learned video compression. ENCORE first employs Complementary Motion Representation (CMR) to decompose aligned RGB-event features into common and modality-specific motion representations. Spatial Energy and Redundancy-Informed Calibration (SERIC) then identifies event-specific responses that are active and novel relative to RGB, suppresses weak or redundant evidence, and predicts a candidate flow correction. Finally, Energy-Aware Routing (EAR) determines where and how strongly the correction should refine the RGB flow. Events serve solely as an auxiliary modality for motion modeling, while RGB remains the only coding and reconstruction target. Experiments on BS-ERGB, HQ-EVFI, and CED demonstrate consistent gains across datasets and GOP lengths. On BS-ERGB, ENCORE achieves up to 20.80% PSNR-RGB and 22.14% MS-SSIM-RGB BD-rate savings, while retaining clear improvements on the other two datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。