让视频模型在极端视角变化下仍能准确识别动作
ViewBridge: Curriculum Knowledge Distillation for Activity View-Invariance Under Extreme Viewpoint Changes
- 用课程学习逐步增加视角难度，配合知识蒸馏保留动作语义
- 在三个数据集上超越现有最优方法，尤其在视角差异大时表现更好
- 适合做真实场景中视角多变的动作识别任务
传统视角不变学习依赖受控的多视角训练数据，但在实际场景中因视角差异大、遮挡严重而表现不佳。本文提出ViewBridge框架，通过引入基于几何的遮挡程度度量，对视频片段按难易排序，采用渐进式课程学习策略，逐步引入更难的视角。同时设计知识蒸馏目标，保留动作核心语义。训练使用多视角数据，推理时仅需单视角未标定视频。在时间关键步定位和细粒度关键步识别两个任务上，于Ego-Exo4D、LEMMA、EPFL-Smart-Kitchen-30三个数据集均超越SOTA方法。
原文摘要 · Abstract (English)
Traditional methods for view-invariant learning rely on controlled multi-view training data with minimal scene clutter. However, they struggle with in-the-wild videos that exhibit extreme viewpoint differences and share little visual content. We introduce ViewBridge, a framework for learning rich video representations in the presence of severe view-occlusions. We introduce a knowledge distillation objective that preserves action-centric semantics, together with a novel curriculum learning procedure that pairs incrementally more challenging views over time, thereby allowing smooth adaptation to extreme viewpoint differences. To sort training video segments for the proposed curriculum, we define a geometry-based metric that reflects their likely occlusion level. While training leverages multi-view data, at inference time, the input is an uncalibrated, single-viewpoint video. Evaluating our approach on two tasks -- temporal keystep grounding and fine-grained keystep recognition -- we outperform SOTA approaches across three datasets (Ego-Exo4D, LEMMA, EPFL-Smart-Kitchen-30). Project page: https://vision.cs.utexas.edu/projects/learning_view_distill/ .
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。