对比两种视频动作识别模型,揭示空间与时间建模的优劣。
Temporal vs. Spatial: Comparing DINOv3 and V-JEPA2 Feature Representations for Video Action Analysis
- DINOv3专注帧内空间特征,V-JEPA2融合时序联合建模。
- DINOv3聚类得分0.31,区分度6.16倍;V-JEPA2方差更低(0.094)。
- 静态动作选DINOv3,复杂动作选V-JEPA2,依任务需求而定。
本研究对两种主流自监督学习架构——DINOv3(独立处理帧,基于空间特征提取)和V-JEPA2(跨视频序列联合建模时序)——在视频动作识别中的表现进行了全面比较。在UCF Sports数据集上,从分类准确率、聚类性能、类内一致性与类间区分度等维度评估特征质量。结果表明:DINOv3在聚类性能(轮廓系数0.31对比0.21)和区分能力(分离比6.16倍)上更优,尤其适用于姿态可辨动作;而V-JEPA2表现出更高的一致性,性能方差仅为0.094,显著低于DINOv3的0.288。进一步分析显示,DINOv3在静态姿态识别中占优,但在依赖运动的动作上表现下降;而V-JEPA2在各类动作中保持均衡表现。研究为视频分析系统的设计选择提供了实证依据。
原文摘要 · Abstract (English)
This study presents a comprehensive comparative analysis of two prominent self-supervised learning architectures for video action recognition: DINOv3, which processes frames independently through spatial feature extraction, and V-JEPA2, which employs joint temporal modeling across video sequences. We evaluate both approaches on the UCF Sports dataset, examining feature quality through multiple dimensions including classification accuracy, clustering performance, intra-class consistency, and inter-class discrimination. Our analysis reveals fundamental architectural trade-offs: DINOv3 achieves superior clustering performance (Silhouette score: 0.31 vs 0.21) and demonstrates exceptional discrimination capability (6.16x separation ratio) particularly for pose-identifiable actions, while V-JEPA2 exhibits consistent reliability across all action types with significantly lower performance variance (0.094 vs 0.288). Through action-specific evaluation, we identify that DINOv3's spatial processing architecture excels at static pose recognition but shows degraded performance on motion-dependent actions, whereas V-JEPA2's temporal modeling provides balanced representation quality across diverse action categories. These findings contribute to the understanding of architectural design choices in video analysis systems and provide empirical guidance for selecting appropriate feature extraction methods based on task requirements and reliability constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。