用语音指令精准追踪视频中被描述的动作目标，兼顾语义理解与运动逻辑。
Motion-Aware Reasoning from Speech to Mask Tracks: Runner-up Solution for the MeViS-Audio Track of the 8th LSVOS Challenge 2026
- 将语音转为结构化约束，涵盖类别、数量、方向等五维信息
- 通过轨迹运动与关系证据排序候选掩码，准确率达第二名
- 融合大模型与条件生成，有效处理无目标场景
语音引导的指代视频目标分割旨在恢复由语音描述的动作对象的掩码轨迹。由于语音提供的是语言指令而非声音证据，解决方案需整合语音识别、以运动为中心的时间定位、掩码跟踪及无目标情况处理。我们提出Speech2MaskTrack，用于第八届LSVOS挑战赛的MeViS-Audio赛道。该方法将语音查询转录并转化为类别、数量、方向、交互角色和时间阶段等结构化约束。SAM3.1生成多个实例轨迹，由TRACE利用完整轨迹运动与关系证据进行排序。冻结的词汇存在门控会抑制基础预测；若门控判断目标存在，则启用全表达条件驱动的SaSaSa2VA轨迹替代SAM3.1掩码。仅当输出为空时，才启动基于GPT的恢复机制，结合查询与掩码级验证再次调用SaSaSa2VA。Speech2MaskTrack在官方排名中获得第二名。
原文摘要 · Abstract (English)
Speech-guided referring video object segmentation aims to recover the mask tracks of objects specified by a spoken motion description. Here, speech carries a linguistic instruction rather than acoustic evidence from a sounding object, so a solution must connect speech recognition, motion-centric temporal grounding, mask tracking, and explicit no-target handling. We introduce Speech2MaskTrack, our approach for the MeViS-Audio track of the 8th LSVOS Challenge. Speech2MaskTrack transcribes the spoken query and compiles it into structured constraints over category, count, direction, interaction role, and temporal phase. SAM3.1 enumerates multiple instance tracks, which TRACE ranks using complete-trajectory motion and relation evidence. A frozen lexical presence gate may suppress the ranked SAM3.1 base prediction. When the gate predicts that a target is present, an available full-expression-conditioned SaSaSa2VA track replaces the SAM3.1 mask. Only outputs that remain empty enter GPT-assisted recovery, which invokes SaSaSa2VA again under query- and mask-level verification. Speech2MaskTrack achieved second place in the official challenge ranking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。