不依赖解码,直接用损坏码流做动作识别,提升鲁棒性。
Bitstream Action Recognition is Byte Modeling

- 双分支字节建模,通过完整码流引导损坏码流的特征对齐
- 在14种方法对比中,对码流损坏的识别准确率最高
- 首次构建大规模损坏码流数据集,适合视频传输场景研究
传统动作识别依赖于码流的正确解码,但存储或传输中的码流损坏会导致严重视觉失真甚至解码失败,影响识别可靠性。为解决这一问题,本文提出新型码流动作识别框架BRACE,采用双分支字节建模结构,将损坏码流与其完整对应版本视为同一动作的两个字节实现。通过完整码流作为稳定锚点,在表示空间内进行嵌入和决策层面的对齐(IARA),并在不可靠锚抑制机制(UAS)下实现无修复的鲁棒表示生成。针对实际中损坏码流稀缺问题,提出四参数真实世界码流损坏模拟器RBCS,可复现传输与存储中的比特翻转和字节丢失错误。基于RBCS构建首个大规模码流动作识别数据集BAR-D,包含BAR-Stanford40和BAR-PPMI子集,覆盖多种损坏类型与强度。在此基础上建立包含14种像素、压缩与码流域方法的大规模基准测试。大量实验表明,BRACE在各类损坏条件下均优于所有对比方法。消融实验进一步验证了RBCS增强与IARA的有效性。
原文摘要 · Abstract (English)
Conventional action recognition typically relies on successful pixel decoding of the bitstream. However, bitstream corruption during storage or transmission may cause severe visual artifacts or even decoding failure, posing a significant challenge to reliable action recognition. Bitstream Action Recognition (BAR) aims to overcome the dependency on decoding and the vulnerability to corruption. In this paper, we propose a novel BAR framework, Bitstream Recognition via Anchoring Corrupted Embeddings (BRACE). BRACE is a dual-branch byte-modeling architecture that treats a corrupted bitstream and its intact counterpart as two byte realizations of the same action. This guides the generation of rich and stable representations for robustness to corruption through Intact-Anchored Representation Alignment (IARA). The intact representation serves as a stable anchor, and the corrupted one is aligned to it at the embedding and decision levels under Unreliable-Anchor Suppression (UAS), entirely in representation space and without repairing the bitstream. To address the scarcity of corrupted bitstreams in practice, we introduce the Real-world Bitstream Corruption Simulator (RBCS), a four-parameter simulator that reproduces bit-flip and byte-loss errors arising in transmission and storage. Building on RBCS, we construct the first large-scale BAR dataset (BAR-D), which comprises the BAR-Stanford40 and BAR-PPMI subsets and spans diverse corruption types and severity levels. Finally, we build a large benchmark on BAR-D involving 14 action recognition methods from the pixel, compressed, and bitstream domains. Extensive experiments demonstrate that BRACE has superior robustness to bitstream corruption than all comparison methods. Ablation studies further validate the effectiveness of the proposed RBCS augmentation and IARA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。