用冻结特征预测用户即将接触的物体和动作,精度达5.13
FROST-STA: Frozen Dense Features for the Ego4D Short-Term Object Interaction Anticipation

- 基于视频与图像的密集特征,通过注意力机制融合多模态信息
- 在Ego4D挑战中取得5.13的Top-5 mAP,排名第二
- 适合对视觉行为预测感兴趣的开发者和研究者
短时预期任务要求系统不仅识别当前场景,还需推断摄像者将接触的物体、后续动作及接触时间。本文介绍FROST-STA,提交至EgoVis 2026 Ego4D短期物体交互预测挑战。模型在每个查询时刻生成包含目标框、名词标签、动词标签、接触时间(TTC)和置信度的结构化假设列表。基于V-JEPA 2.1评估协议,采用物体中心解码、多头预测和提交优化训练策略。固定ViT-G骨干网络,提取两个密集标记流:查询前384像素缩放片段的视频标记,以及最后高分辨率帧的图像标记。通过注意力探针与帧引导的时间池化模块,将片段表示映射到最终帧空间并融合图像特征。融合后的特征由类Faster R-CNN的STA头部解码,估计框偏移、名词、动词、TTC值与交互质量。最终榜单提交使用官方训练集加允许的验证标注,训练25轮,融合第15至25轮共8个头与检查点。模型在官方测试服务器上取得5.13的Overall Top-5 mAP,位列第二,表明冻结的密集图文特征可作为物体级交互预测的强大基础。
原文摘要 · Abstract (English)
Short-term anticipation in egocentric video requires more than recognizing the current scene: a system must infer which object the camera wearer will contact, which action will follow, and how soon the contact will happen. This report describes FROST-STA, our submission to the Ego4D Short-Term Object Interaction Anticipation (STA) Challenge at EgoVis 2026. For each query time, the model produces a ranked set of structured hypotheses containing an active-object box, noun label, verb label, time-to-contact (TTC), and confidence. FROST-STA builds on the V-JEPA 2.1 STA evaluation protocol, but adapts it to the challenge by using object-centric decoding, multi-head prediction, and a submission-oriented training and ensembling recipe. We keep the V-JEPA 2.1 ViT-G backbone fixed and extract two dense token streams: video tokens from a short clip resized to 384 pixels before the query, and image tokens from the last observed high-resolution frame. A compact alignment module, consisting of an attentive probe and frame-guided temporal pooling, maps the clip representation onto the spatial reference of the final frame before fusing it with image features. The fused maps are decoded by Faster R-CNN-style STA heads that estimate box offsets, nouns, verbs, TTC values, and interaction quality. For the final leaderboard entry, we train for 25 epochs with the official training split plus additional permitted validation annotations, and combine predictions across eight heads and checkpoints from epochs 15-25. FROST-STA obtains 5.13 Overall Top-5 mAP on the official test server, ranking second in the challenge and showing that frozen dense image-video features can serve as a strong basis for object-level interaction forecasting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。