用关键帧+SAM3实现零训练视频目标分割，支持自然语言指令
Key-Frame Reasoning with SAM3: Third Place Solution for the MeViS-Text Track of the 8th LSVOS Challenge
- 先用Gemini拆解语言指令找关键帧，再用SAM3生成像素级掩码
- 在单张RTX 4090上运行，无需训练，测试集综合得分0.8566
- 适合需要快速部署、不依赖训练数据的视频理解任务
本文提出一种两阶段、免训练的解决方案，用于第八届LSVOS挑战赛的MeViS-Text赛道。该任务要求模型根据自然语言表达，在整个视频中定位并分割指定对象。此类表达常依赖时间线索，如动作、交互、方向和相对位置。第一阶段通过API调用Gemini-3.1 Pro，将视频级事件分解为实例级目标，为每个目标选定关键帧，并生成与该帧对齐的判别性描述。第二阶段，SAM3-agent在选定帧上生成像素级种子掩码，SAM3视频追踪器双向传播掩码至全视频。有效实例独立定位并传播，最终合并帧级掩码。所有SAM3处理均在单张NVIDIA GeForce RTX 4090上完成，无需任务特定训练或模型集成。方法在挑战赛测试集上排名第三，获得J&F、J、F、N-acc.、T-acc.和Final分数分别为0.761、0.7367、0.7852、0.8333、0.9755和0.856593。
原文摘要 · Abstract (English)
This report presents a two-stage, training-free solution for the MeViS-Text track of the 8th LSVOS Challenge. The task requires a model to localize and segment the object specified by a natural-language expression throughout a video. Such expressions often depend on temporal cues, including actions, interactions, directions, and relative positions. Our first stage uses Gemini-3.1 Pro via API to decompose a video-level event into instance-level targets, select a key frame for each target, and generate a discriminative description aligned with that frame. In the second stage, SAM3-agent produces a pixel-level seed mask on the selected frame, and the SAM3 video tracker propagates the mask bidirectionally through the video. Valid instances are grounded and propagated independently before their frame-wise masks are merged. All local SAM3 processing runs on a single NVIDIA GeForce RTX 4090 without task-specific training or model ensembling. Our method ranked third on the challenge test set, obtaining J&F, J, F, N-acc., T-acc., and Final scores of 0.761, 0.7367, 0.7852, 0.8333, 0.9755, and 0.856593, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。