为机器视觉任务设计可变换的视频特征压缩框架，节省带宽并支持多任务。
TransVFC: A Transformable Video Feature Compression Framework for Machines
- 基于编码-转换架构，通过时序预测与感知引导编码减少冗余。
- 在多个下游任务上实现高率-任务性能，支持不同粒度任务。
- 新增任务只需训练轻量模块，无需重训主干网络，适合多任务场景。
当前越来越多的视频传输目标是下游机器视觉任务而非人眼感知。尽管广泛部署的人类视觉系统（HVS）导向视频编码标准如H.265/HEVC和H.264/AVC效率较高，但并非视频编码用于机器（VCM）场景的最优方案，导致不必要的码率消耗。学术和技术界在VCM领域已提出多种策略，但在多任务场景下的适应性仍存明显局限。为此，我们提出可变换视频特征压缩（TransVFC）框架，提供一种先压缩后传输的解决方案，包含视频特征编解码器和特征空间变换（FST）模块。具体而言，编解码器通过基于方案的帧间预测模块压缩视频特征的时间冗余；再利用感知引导的条件编码最小化空间冗余，并使重建特征更符合下游机器感知。随后，通过FST模块将重建特征映射至新特征空间以适配不同下游任务。新增任务仅需训练一个轻量级FST模块，避免重训上游编解码器及下游任务网络。实验表明，TransVFC在多种粒度的下游任务中均实现了优异的率-任务性能。我们期望本工作能为多任务场景下的视频特征压缩提供重要参考。代码见：https://github.com/Ws-Syx/TransVFC。
原文摘要 · Abstract (English)
Nowadays, more and more video transmissions primarily aim at downstream machine vision tasks rather than humans. While widely deployed Human Visual System (HVS) oriented video coding standards like H.265/HEVC and H.264/AVC are efficient, they are not the optimal approaches for Video Coding for Machines (VCM) scenarios, leading to unnecessary bitrate expenditure. The academic and technical exploration within the VCM domain has led to the development of several strategies, and yet, conspicuous limitations remain in their adaptability for multi-task scenarios. To address the challenge, we propose a Transformable Video Feature Compression (TransVFC) framework. It offers a compress-then-transfer solution and includes a video feature codec and Feature Space Transform (FST) modules. In particular, the temporal redundancy of video features is squeezed by the codec through the scheme-based inter-prediction module. Then, the codec implements perception-guided conditional coding to minimize spatial redundancy and help the reconstructed features align with downstream machine perception.After that, the reconstructed features are transferred to new feature spaces for diverse downstream tasks by FST modules. To accommodate a new downstream task, it only requires training one lightweight FST module, avoiding retraining and redeploying the upstream codec and downstream task networks. Experiments show that TransVFC achieves high rate-task performance for diverse tasks of different granularities. We expect our work can provide valuable insights for video feature compression in multi-task scenarios. The codes are at https://github.com/Ws-Syx/TransVFC.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。