提出轻量级二值化视频抠图模型,显著提升边缘设备上的推理效率与精度。
BiVM: Accurate Binarized Neural Network for Efficient Video Matting
- 设计可弹性扩展的二值化结构与进化式拓扑,增强编码器特征提取能力。
- 通过掩码稀疏解码器中间特征,减少冗余计算,聚焦细节区域。
- 引入信息引导的模仿框架,精准迁移全精度模型的有用表征,适合移动端部署。
面向实时视频抠图在边缘设备上的计算瓶颈,现有深度神经网络受限于高算力需求。二值化作为压缩手段,虽具1比特参数和高效位运算优势,但因编码器退化与解码器冗余导致精度与效率双重下降。基于信息瓶颈理论分析,问题根源在于中间特征中预测相关信 息的退化及无关区域的冗余计算。本文提出BiVM——一种高精度且资源高效的二值化视频抠图网络。首先,设计具有弹性捷径与可演化拓扑的二值化计算模块,构建高质量编码器以提取输入视频有效表征。其次,通过掩码剔除解码器中间特征中的同质区域,使计算聚焦于细节丰富部分,降低推理负担。此外,构建局部二值化感知的模仿框架,采用信息引导策略,确保全精度模型中的关键表征被准确、完整地迁移。大量实验表明,BiVM显著优于现有二值化视频抠图方法,包括当前最优二值化技术。在计算与存储成本上分别实现14.3倍和21.6倍压缩。同时在ARM CPU硬件上进行了评估,验证了其实际部署可行性。
原文摘要 · Abstract (English)
Deep neural networks for real-time video matting suffer significant computational limitations on edge devices, hindering their adoption in widespread applications such as online conferences and short-form video production. Binarization emerges as one of the most common compression approaches with compact 1-bit parameters and efficient bitwise operations. However, accuracy and efficiency limitations exist in the binarized video matting network due to its degenerated encoder and redundant decoder. Following a theoretical analysis based on the information bottleneck principle, the limitations are mainly caused by the degradation of prediction-relevant information in the intermediate features and the redundant computation in prediction-irrelevant areas. We present BiVM, an accurate and resource-efficient Binarized neural network for Video Matting. First, we present a series of binarized computation structures with elastic shortcuts and evolvable topologies, enabling the constructed encoder backbone to extract high-quality representation from input videos for accurate prediction. Second, we sparse the intermediate feature of the binarized decoder by masking homogeneous parts, allowing the decoder to focus on representation with diverse details while alleviating the computation burden for efficient inference. Furthermore, we construct a localized binarization-aware mimicking framework with the information-guided strategy, prompting matting-related representation in full-precision counterparts to be accurately and fully utilized. Comprehensive experiments show that the proposed BiVM surpasses alternative binarized video matting networks, including state-of-the-art (SOTA) binarization methods, by a substantial margin. Moreover, our BiVM achieves significant savings of 14.3x and 21.6x in computation and storage costs, respectively. We also evaluate BiVM on ARM CPU hardware.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。