用可微局部对齐提升视频表征，让模型更懂动作时序细节。
Self-Supervised Contrastive Learning for Videos using Differentiable Local Alignment
- 通过可微局部对齐损失捕捉视频帧间局部时序关系。
- 在动作识别任务中超越现有最先进方法，准确率显著提升。
- 适合需要精准时序理解的视频分析场景，如行为识别。
鲁棒的帧级嵌入对视频分析与理解至关重要。本文提出一种基于时序视频序列对齐的自监督表示学习方法。框架采用基于Transformer的编码器提取帧级特征，并利用这些特征寻找视频序列间的最优对齐路径。提出新颖的局部对齐对比（LAC）损失，结合可微局部对齐损失以捕捉局部时序依赖，以及对比损失以增强判别性学习。与以往关注序列对全局时序排序的工作不同，LAC鼓励识别最佳得分的子序列对齐。该方法采用可微的Smith-Waterman（SW）仿射算法，其参数在训练过程中动态学习，使模型能自适应调整时间间隙惩罚长度。实验表明，所学表征在动作识别任务上优于现有最先进方法。
原文摘要 · Abstract (English)
Robust frame-wise embeddings are essential to perform video analysis and understanding tasks. We present a self-supervised method for representation learning based on aligning temporal video sequences. Our framework uses a transformer-based encoder to extract frame-level features and leverages them to find the optimal alignment path between video sequences. We introduce the novel Local-Alignment Contrastive (LAC) loss, which combines a differentiable local alignment loss to capture local temporal dependencies with a contrastive loss to enhance discriminative learning. Prior works on video alignment have focused on using global temporal ordering across sequence pairs, whereas our loss encourages identifying the best-scoring subsequence alignment. LAC uses the differentiable Smith-Waterman (SW) affine method, which features a flexible parameterization learned through the training phase, enabling the model to adjust the temporal gap penalty length dynamically. Evaluations show that our learned representations outperform existing state-of-the-art approaches on action recognition tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。