直接在压缩码流上识别动作,抗损坏且保护隐私
ByteAction: Byte-space Action Recognition Foundation Model

- 用双重码流视图与字节级增强学习跨区域依赖
- 在斯坦福40等数据集上对损坏码流识别率超基准
- 适合需要隐私保护或传输不稳定的视频分析场景
字节空间动作识别(BAR)旨在直接从压缩图像码流中识别人类动作,无需解码像素。通过完全在字节空间操作,BAR天然不受文件完整性与像素重建影响,适用于隐私敏感场景并具备抗码流损坏能力。本文提出ByteAction,一种BAR基础模型,在受损码流上实现高精度动作识别。其采用双视图字节级识别框架,通过比特流模式增强(BPA)构建弱/强损坏码流视图,并以共享ByteFormer骨干网络编码。模型同时优化分类与损坏一致性目标。BPA将一维字节序列转为二维矩阵并进行区域擦除,促进模型学习鲁棒的跨区域字节依赖;提出损坏一致性训练策略,通过双向KL散度约束模型在不同损坏程度下保持稳定预测。在斯坦福40、PPMI及PASCAL VOC 2012 Action的图像码流上实验表明,ByteAction在所有场景下均达到最先进的抗损坏性能,同时保持与完整码流相当的识别精度。
原文摘要 · Abstract (English)
Byte-space Action Recognition (BAR) aims to recognize human actions directly from compressed image bitstreams without any pixel decoding. By operating entirely in byte space, BAR is inherently independent of file integrity and pixel-level reconstruction, making it naturally applicable to privacy-sensitive scenarios and robust against bitstream corruption. In this paper, we propose ByteAction, a BAR foundation model that achieves accurate action recognition on corrupted image bitstreams. ByteAction follows a dual-view byte-level recognition framework. It constructs weakly and strongly corrupted bitstream views, which are augmented by Bitstream Pattern Augmentation (BPA) and encoded with a shared ByteFormer backbone. The model is optimized with both classification and corruption consistency objectives. Specifically, we propose Bitstream Pattern Augmentation (BPA), which reshapes one-dimensional byte sequences into two-dimensional byte matrix and applies region-level erasure to encourage the model to learn robust cross-region byte dependencies. We further propose a Corruption Consistency Training strategy that constrains the model to maintain stable predictions across different corruption severities through bidirectional KL divergence. Experiments on the image bitstream from Stanford40, PPMI, and PASCAL VOC 2012 Action demonstrate that ByteAction achieves state-of-the-art corruption robustness across all scenarios while maintaining competitive intact bitstream performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。