提出CMeRT模型，提升在线动作检测的上下文感知与未来预测能力。
Context-Enhanced Memory-Refined Transformer for Online Action Detection
- 用近邻上下文增强编码器，优化帧表示
- 通过近未来生成提升解码器性能，减少训练推理差异
- 在多个数据集上达到当前最优，适合视频理解研究者
在线动作检测（OAD）利用历史观测实时识别流式视频中的动作。现有方法通过短时和长时记忆建模过去，并借助未来预期补偿缺失信息。我们发现现有方法存在训练与推理阶段短时记忆长度不一致的问题，影响学习效果。为此，提出上下文增强的记忆精炼变压器（CMeRT）。CMeRT引入上下文增强编码器，利用更早的近期上下文改进帧表征；同时设计记忆精炼解码器，利用近未来帧生成来提升性能。CMeRT在THUMOS'14、CrossTask和EPIC-Kitchens-100数据集上均取得在线检测与未来预测的最优结果。
原文摘要 · Abstract (English)
Online Action Detection (OAD) detects actions in streaming videos using past observations. State-of-the-art OAD approaches model past observations and their interactions with an anticipated future. The past is encoded using short- and long-term memories to capture immediate and long-range dependencies, while anticipation compensates for missing future context. We identify a training-inference discrepancy in existing OAD methods that hinders learning effectiveness. The training uses varying lengths of short-term memory, while inference relies on a full-length short-term memory. As a remedy, we propose a Context-enhanced Memory-Refined Transformer (CMeRT). CMeRT introduces a context-enhanced encoder to improve frame representations using additional near-past context. It also features a memory-refined decoder to leverage near-future generation to enhance performance. CMeRT achieves state-of-the-art in online detection and anticipation on THUMOS'14, CrossTask, and EPIC-Kitchens-100.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。