用未来动作引导注意力,提升视频动作预测的泛化能力
Action-Guided Attention for Video Action Anticipation
- 以预测动作序列作为查询和键,指导注意力聚焦关键历史片段
- 在EPIC-Kitchens-100上实现优异的跨集泛化性能,验证模型鲁棒性
- 支持训练后分析,可揭示模型捕捉的动作依赖与反事实推理逻辑
视频动作预测极具挑战性,因观察帧仅提供过往活动的证据,需推断潜在意图以预测未来动作。现有基于Transformer的方法依赖像素表示的点积注意力,缺乏建模视频序列所需的高层语义,易过拟合于过去帧中的显式视觉线索,限制对潜在意图的捕捉,降低对未见样本的泛化能力。为此,我们提出动作引导注意力(AGA),该机制显式利用预测动作序列作为查询与键,引导序列建模。AGA使注意力模块能根据未来动作强调相关历史时刻,并通过专用门控函数将此信息与当前帧嵌入融合。该设计支持训练后分析,可考察模型从训练数据中学到的知识。在广泛使用的EPIC-Kitchens-100基准上的实验表明,AGA在验证集到未见测试集间具有良好的泛化能力。训练后分析还可进一步揭示模型捕捉的动作依赖关系及内部化的反事实证据,为预测过程提供透明可解释的洞察。
原文摘要 · Abstract (English)
Anticipating future actions in videos is challenging, as the observed frames provide only evidence of past activities, requiring the inference of latent intentions to predict upcoming actions. Existing transformer-based approaches, which rely on dot-product attention over pixel representations, often lack the high-level semantics necessary to model video sequences for effective action anticipation. As a result, these methods tend to overfit to explicit visual cues present in the past frames, limiting their ability to capture underlying intentions and degrading generalization to unseen samples. To address this, we propose Action-Guided Attention (AGA), an attention mechanism that explicitly leverages predicted action sequences as queries and keys to guide sequence modeling. Our approach fosters the attention module to emphasize relevant moments from the past based on the upcoming activity and combine this information with the current frame embedding via a dedicated gating function. The design of AGA enables post-training analysis of the knowledge discovered from the training set. Experiments on the widely adopted EPIC-Kitchens-100 benchmark demonstrate that AGA generalizes well from validation to unseen test sets. Post-training analysis can further examine the action dependencies captured by the model and the counterfactual evidence it has internalized, offering transparent and interpretable insights into its anticipative predictions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。