用双流对齐提升动作分割精度,融合帧与动作特征增强识别能力。
Dual-Stream Alignment for Action Segmentation
- 设计双流结构,分别处理帧级和动作级特征,通过注意力机制融合
- 在多个数据集上达到最新性能,显著优于现有方法
- 首次引入量子-经典混合框架,适合视频分析与动作识别研究者
动作分割旨在识别连续视频中特定动作的发生时间与位置。现有方法多采用单流模型建模时空序列,而近期趋势转向双流方法,利用动作级特征提升分割性能。本文提出双流对齐网络(DSA Net),引入第二路学习到的动作特征,以捕捉动作及动作过渡线索。两流间通过时序上下文(TC)模块实现信息交互,该模块采用交叉注意力与基于量子的动作引导调制(Q-ActGM)融合互补信息。为实现两流特征空间对齐,提出双流对齐损失,包含关系一致性、跨层次对比和循环一致性重建三部分。在GTEA、Breakfast、50Salads和EgoProcel等多个基准数据集上验证了模型有效性,大量消融实验证明各组件贡献。DSA Net达到当前最优性能。
原文摘要 · Abstract (English)
Action segmentation is a challenging yet active research area that involves identifying when and where specific actions occur in continuous video streams. Most existing work has focused on single-stream approaches that model the spatio-temporal aspects of frame sequences. However, recent research has shifted toward two-stream methods that learn action-wise features to enhance action segmentation performance. In this work, we propose the Dual-Stream Alignment Network (DSA Net) and investigate the impact of incorporating a second stream of learned action features to guide segmentation by capturing both action and action-transition cues. Communication between the two streams is facilitated by a Temporal Context (TC) block, which fuses complementary information using cross-attention and Quantum-based Action-Guided Modulation (Q-ActGM), enhancing the expressive power of the fused features. To the best of our knowledge, this is the first study to introduce a hybrid quantum-classical machine learning framework for action segmentation. Our primary objective is for the two streams (frame-wise and action-wise) to learn a shared feature space through feature alignment. This is encouraged by the proposed Dual-Stream Alignment Loss, which comprises three components: relational consistency, cross-level contrastive, and cycle-consistency reconstruction losses. Following prior work, we evaluate DSA Net on several diverse benchmark datasets: GTEA, Breakfast, 50Salads, and EgoProcel. We further demonstrate the effectiveness of each component through extensive ablation studies. Notably, DSA Net achieves state-of-the-art performance, significantly outperforming existing
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。