通过时间约束建模，提升长时动作预测的连贯性与准确性。
Temporal Context Consistency Above All: Enhancing Long-Term Anticipation by Learning and Enforcing Temporal Constraints
- 引入双向上下文正则化模块，保证相邻片段时间一致性。
- 学习动作转移概率矩阵，在全时长上全局优化预测序列。
- 专用编码器增强初始片段理解，适合视频动作预测研究者。
本文提出一种长时动作预测（LTA）方法，旨在基于初始未修剪视频片段预测未来动作标签及其持续时间。基于编码器-解码器架构并采用并行解码，主要贡献有二：其一，在解码器顶部引入双向动作上下文正则化模块，确保时间相邻片段间上下文一致性；其二，从已分类片段中学习动作转移矩阵，建模动作间转移概率，并在完整预测区间内进行全局优化。此外，使用专用编码器提升观察区间内的动作分割质量，增强对过去状态的理解。在EpicKitchen-55、EGTEA+、50Salads和Breakfast四个基准数据集上验证，性能优于或媲美现有先进方法，包括假设输入为剪辑后视频的概率模型及基于大语言模型的方法。代码将在录用后公开。
原文摘要 · Abstract (English)
This paper proposes a method for long-term action anticipation (LTA), the task of predicting action labels and their duration in a video given the observation of an initial untrimmed video interval. We build on an encoder-decoder architecture with parallel decoding and make two key contributions. First, we introduce a bi-directional action context regularizer module on the top of the decoder that ensures temporal context coherence in temporally adjacent segments. Second, we learn from classified segments a transition matrix that models the probability of transitioning from one action to another and the sequence is optimized globally over the full prediction interval. In addition, we use a specialized encoder for the task of action segmentation to increase the quality of the predictions in the observation interval at inference time, leading to a better understanding of the past. We validate our methods on four benchmark datasets for LTA, the EpicKitchen-55, EGTEA+, 50Salads and Breakfast demonstrating superior or comparable performance to state-of-the-art methods, including probabilistic models and also those based on Large Language Models, that assume trimmed video as input. The code will be released upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。