提出混合视差补偿方法,提升立体视频压缩效率。
Neural Stereo Video Compression with Hybrid Disparity Compensation
- 结合显式位移与隐式注意力,优化跨视角特征对齐。
- 在KITTI和Nagoya数据集上显著优于传统与神经方法。
- 适合研究立体视频编码与多视角视觉任务的开发者。
视差补偿是立体视频压缩(SVC)中利用跨视角冗余的核心策略,可分为显式水平位移与隐式交叉注意力两类。本文提出一种混合视差补偿(HDC)策略:首先通过融合水平偏移的跨视角特征生成相似性图,捕捉像素位移信息;再将该图归一化为“显式逐像素注意力分数”,用于后续隐式交叉注意力对齐操作,从而更全面地建模视差信息。基于HDC,我们设计了一种端到端优化的神经立体视频压缩框架,将HDC模块融入关键编码步骤,包括跨视角特征提取与重建(HDC-FER)及跨视角熵建模(HDC-EM)。在KITTI 2012、KITTI 2015和Nagoya等基准测试中,涵盖自动驾驶与通用场景,实验结果表明本框架在性能上超越现有神经与传统SVC方法。
原文摘要 · Abstract (English)
Disparity compensation represents the primary strategy in stereo video compression (SVC) for exploiting cross-view redundancy. These mechanisms can be broadly categorized into two types: one that employs explicit horizontal shifting, and another that utilizes an implicit cross-attention mechanism to reduce cross-view disparity redundancy. In this work, we propose a hybrid disparity compensation (HDC) strategy that leverages explicit pixel displacement as a robust prior feature to simplify optimization and perform implicit cross-attention mechanisms for subsequent warping operations, thereby capturing a broader range of disparity information. Specifically, HDC first computes a similarity map by fusing the horizontally shifted cross-view features to capture pixel displacement information. This similarity map is then normalized into an "explicit pixel-wise attention score" to perform the cross-attention mechanism, implicitly aligning features from one view to another. Building upon HDC, we introduce a novel end-to-end optimized neural stereo video compression framework, which integrates HDC-based modules into key coding operations, including cross-view feature extraction and reconstruction (HDC-FER) and cross-view entropy modeling (HDC-EM). Extensive experiments on SVC benchmarks, including KITTI 2012, KITTI 2015, and Nagoya, which cover both autonomous driving and general scenes, demonstrate that our framework outperforms both neural and traditional SVC methodologies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。