arXiv:2606.23256cs.CVcs.AI2026-06

P-JEPA让模型能高效理解30分钟长的步骤视频,突破了注意力机制的瓶颈。

P-JEPA: Procedural Video Representation Learning via Joint Embedding Predictive Architecture

论文配图:P-JEPA: Procedural Video Representation Learning via Joint Embedding Predictive Architecture
图 1 · 摘自论文原文
  • 将长视频建模为帧对齐动作空间,用掩码向量预测降低计算复杂度
  • 在多个数据集上实现更优的动作分类与分割,且推理速度快、参数少
  • 适合需要长期依赖理解的智能助手、自动化流程分析等场景

随着具身智能平台的发展,对复杂多步任务的程序化视频表征学习需求日益增长。尽管大规模隐变量预测训练使视频基础模型能捕捉动态特征,支持活动理解、时空定位和预测控制等下游任务,但现有方法难以处理具有长程依赖的动作序列,因自注意力存在二次复杂度。例如,灶具开关动作视觉相似却位置不同。本文提出一种不依赖主干网络的方法,将长时视频表征问题转化为帧对齐的动作空间,通过预测聚合的掩码隐向量实现建模。该方法使所提的程序化联合嵌入预测架构(P-JEPA)可处理超30分钟视频,有效实现长流程理解。我们在EgoExo4D、EgoProceL和Assembly101数据集上使用VJEPA2.1、TSM和I3D提取特征进行评估,结果表明,P-JEPA在线性可分性、流式推理和时序动作分割方面均表现优异,在EgoExo4D细粒度动作分类上达到当前最优性能,参数量仅为基于LLM方法的十分之一,且可实时运行。

原文摘要 · Abstract (English)

The increasing maturity of embodied AI platforms has driven a growing interest in procedural video representation learning to support intelligent assistance systems for complex, multi-step tasks. Leveraging large-scale latent predictive training, video foundation models capture video dynamics, enabling downstream tasks such as activity understanding, spatiotemporal localization, and predictive control. However, procedural videos include actions with long-range dependencies that these models do not support, due to the quadratic complexity of self-attention. Distinct actions, for example, may be visually similar despite appearing at different points in the procedure, such as turning the stove on versus off. Here, we propose a backbone-agnostic approach that learns long-duration video representations by reducing the problem to a dense, frame-aligned action space and predicting pooled masked latent vectors. This approach allows our Procedural Joint Embedding Predictive Architecture (P-JEPA) to ingest videos over 30 minutes long, enabling effective long-form understanding of procedural steps. We evaluate P-JEPA using features extracted with VJEPA2.1, TSM, and I3D over the EgoExo4D, EgoProceL, and Assembly101 datasets, finding that it consistently improves linear separability, streaming inference, and temporal action segmentation performance, achieving state-of-the-art results on EgoExo4D fine-grained action classification while using an order of magnitude fewer parameters than LLM-based methods and running in real time.

视频表征长视频理解自回归建模具身智能

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。