提出新方法分离视频的视角不变与视角变化特征，提升跨视角动作识别性能。
PRISM: Predictive Recomposition via Semantic Latent Decomposition for View-invariant Video Representation Learning
- 通过语义潜在分解，将视频拆分为视角不变和视角变化两部分。
- 在EgoExo4D等数据集上达到当前最优，零样本下超越域内模型。
- 适合做跨视角动作识别或视频表征学习的研究者参考。
跨视角视频表征学习旨在捕捉尽管在第一人称与第三人称视频间外观变化显著时仍保持一致的动作语义。然而，现有方法将每个视频编码为单一嵌入，导致视角不变与视角变化语义在共现情况下不可避免地纠缠——我们证明这种失效模式即使在专为视角不变性训练的方法中也普遍存在。核心洞察在于：当视角不变特征能与任意视角变化特征充分重组并保留各自独立语义时，才算真正解耦。基于此，我们提出PRISM，将视频分解为视角不变和视角变化潜在表示，并在语言监督下进行重组，以促进两者的清晰分离。PRISM在EgoExo4D、EgoExoLearn、AE2数据集上取得当前最优结果，甚至在零样本设置下超越域内模型。代码已公开于https://github.com/litcoderr/prism。
原文摘要 · Abstract (English)
Cross-view video representation learning aims to capture viewpoint-invariant action semantics despite substantial appearance changes across egocentric and exocentric videos. However, existing methods encode each video as a unified embedding, where view-invariant and view-variant semantics inevitably entangle under co-occurrences - a failure mode we show persists even in cross-view methods explicitly trained for view-invariance. Our key insight is that a view-invariant feature is truly disentangled when it can be sufficiently recomposed with an arbitrary view-variant feature while preserving their independent semantics. Building on this, we propose PRISM, that decomposes video into view-invariant and view-variant latents and recompose them under language supervision encouraging clean decomposition of the two streams. PRISM achieves state-of-the-art results on EgoExo4D, EgoExoLearn, AE2, even surpassing in-domain models under zero-shot setting. Code is available at https://github.com/litcoderr/prism.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。