VISTA模型预测未来1秒内人与物体的交互,准确率领先。
VISTA: Technical Report for the Ego4D Short-Term Object Interaction Anticipation at EgoVis 2026

- 结合物体检测与短时序上下文,用特征调制融合视觉与时间信息。
- 在Ego4D挑战赛中达成最高精度,对齐框、类别和接触时间预测均最优。
- 适合关注第一视角视频预测、人机交互建模的研究者使用。
我们提出VISTA,一种基于V-JEPA的集成式快速静态时序预测框架,用于在EgoVis 2026的Ego4D短时物体交互预测(STA)挑战中进行预测。给定第一人称视频的时间戳,任务要求预测下一次人-物交互,包括未来主动物体的边界框、名词类别、动词类别、接触时间及置信度。VISTA采用StillFast风格设计,结合以物体为中心的空间检测与短时序上下文。具体地,使用在COCO上预训练的Faster R-CNN ResNet-50 FPN检测器从最后一帧高分辨率图像生成物体候选区域;同时,冻结的V-JEPA 2.1时序分支从观测视频中提取片段级第一人称上下文。时序表示通过特征调制与ROI级上下文融合注入检测路径。融合后的候选区域特征输入多头STA预测器,完成框精修、名词分类、动词分类、接触时间回归与交互置信度估计。最终提交版本进一步集成互补预测以提升鲁棒性。实验结果表明,VISTA在官方评测服务器上获得第一名。代码将发布于https://github.com/CorrineQiu/VISTA。
原文摘要 · Abstract (English)
We propose VISTA, a V-JEPA Integrated StillFast Temporal Anticipator for the Ego4D Short-Term Object Interaction Anticipation (STA) Challenge at EgoVis 2026. Given an egocentric video timestamp, the task requires anticipating the next human-object interaction, including the future active object's bounding box, noun category, verb category, time-to-contact, and confidence score. VISTA follows a StillFast-style design that combines object-centric spatial detection with short-horizon temporal context. Specifically, a COCO-pretrained Faster R-CNN ResNet-50 FPN detector generates object proposals from the last observed high-resolution frame, while a frozen V-JEPA 2.1 temporal branch extracts clip-level egocentric context from the observed video. The temporal representation is injected into the detection pathway through feature modulation and ROI-level context fusion. The fused proposal features are then passed to multi-head STA predictors for box refinement, noun classification, verb classification, time-to-contact regression, and interaction confidence estimation. For the final submission, we further ensemble complementary predictions to improve robustness. Experimental results on the official challenge server show that VISTA achieves first place in the EgoVis 2026 Ego4D STA Challenge. Our code will be released at https://github.com/CorrineQiu/VISTA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。