零样本生成厨房视频场景图，让模型稳定识别动态物体。
SAMJAM: Zero-Shot Video Scene Graph Generation for Egocentric Kitchen Videos
- 用SAM2跟踪+Gemini语义理解，跨帧保持物体一致
- 在EPIC-KITCHENS数据集上比Gemini高8.33%平均召回率
- 适合做零样本视频理解的科研人员和开发者
视频场景图生成（VidSGG）是理解动态厨房环境的重要任务。现有模型需大量训练才能生成场景图。近期视觉语言模型（VLM）和视觉基础模型（VFM）在多种任务中展现出出色的零样本能力。然而，如Gemini这类VLM在处理VidSGG时难以维持跨帧的物体身份一致性。为此，我们提出SAMJAM，一种结合SAM2时间跟踪与Gemini语义理解的零样本流水线。SAM2通过更精确的边界框提升Gemini的物体定位能力。方法流程为：先用Gemini生成每帧的场景图；再通过匹配算法将场景图中的物体与SAM2生成或传播的掩码对齐，构建出动态环境下的时序一致场景图；最后对后续每一帧重复此过程。实证表明，SAMJAM在EPIC-KITCHENS和EPIC-KITCHENS-100数据集上相较Gemini提升了8.33%的平均召回率。
原文摘要 · Abstract (English)
Video Scene Graph Generation (VidSGG) is an important topic in understanding dynamic kitchen environments. Current models for VidSGG require extensive training to produce scene graphs. Recently, Vision Language Models (VLM) and Vision Foundation Models (VFM) have demonstrated impressive zero-shot capabilities in a variety of tasks. However, VLMs like Gemini struggle with the dynamics for VidSGG, failing to maintain stable object identities across frames. To overcome this limitation, we propose SAMJAM, a zero-shot pipeline that combines SAM2's temporal tracking with Gemini's semantic understanding. SAM2 also improves upon Gemini's object grounding by producing more accurate bounding boxes. In our method, we first prompt Gemini to generate a frame-level scene graph. Then, we employ a matching algorithm to map each object in the scene graph with a SAM2-generated or SAM2-propagated mask, producing a temporally-consistent scene graph in dynamic environments. Finally, we repeat this process again in each of the following frames. We empirically demonstrate that SAMJAM outperforms Gemini by 8.33% in mean recall on the EPIC-KITCHENS and EPIC-KITCHENS-100 datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。