用掩码变压器简化多描述视频编码，提升抗网络波动能力
Robust Multiple Description Neural Video Codec with Masked Transformer for Dynamic and Noisy Networks
- 用双向掩码变压器建模视频隐向量时序依赖，替代传统运动预测
- 在动态噪声网络下，丢包恢复性能超越现有最优神经编码器
- 适合对传输鲁棒性要求高的实时视频应用，如远程医疗、直播
多描述编码（MDC）是一种极具潜力的抗错源编码方法，特别适用于具有多条（但嘈杂且不可靠）路径的动态网络。然而，传统MDC视频编码器存在架构复杂、可扩展性差、容错能力有限和压缩效率低等问题，导致其未被广泛应用。受神经视频编码潜力启发，本文重新思考MDC设计，提出新型MDC视频编码器NeuralMDC。该方法利用双向掩码变换器进行掩码标记预测训练，大幅简化了MDC编码器设计。编码时，将每帧视频转化为隐向量表示并拆分为多个含相关性的描述；不使用运动预测与插值操作，而是通过双向掩码变换器建模隐向量的空间-时序依赖，并基于历史信息预测当前表示分布。该分布用于独立熵编码各描述，并推断可能丢失的标记。大量实验表明，NeuralMDC在压缩效率损失极小的前提下，实现了最先进的容错能力，显著优于现有基于残差编码的抗错神经视频编码器。
原文摘要 · Abstract (English)
Multiple Description Coding (MDC) is a promising error-resilient source coding method that is particularly suitable for dynamic networks with multiple (yet noisy and unreliable) paths. However, conventional MDC video codecs suffer from cumbersome architectures, poor scalability, limited loss resilience, and lower compression efficiency. As a result, MDC has never been widely adopted. Inspired by the potential of neural video codecs, this paper rethinks MDC design. We propose a novel MDC video codec, NeuralMDC, demonstrating how bidirectional transformers trained for masked token prediction can vastly simplify the design of MDC video codec. To compress a video, NeuralMDC starts by tokenizing each frame into its latent representation and then splits the latent tokens to create multiple descriptions containing correlated information. Instead of using motion prediction and warping operations, NeuralMDC trains a bidirectional masked transformer to model the spatial-temporal dependencies of latent representations and predict the distribution of the current representation based on the past. The predicted distribution is used to independently entropy code each description and infer any potentially lost tokens. Extensive experiments demonstrate NeuralMDC achieves state-of-the-art loss resilience with minimal sacrifices in compression efficiency, significantly outperforming the best existing residual-coding-based error-resilient neural video codec.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。