修复训练推理不一致问题，显著提升视频目标分割性能
Sa2VA-i: Improving Sa2VA Results with Consistent Training and Inference
- 修正训练与推理流程差异，使模型发挥真实潜力
- 在多个基准上提升最高达11.6的J&F指标
- 小模型性能媲美大模型，适合追求效率的研究者
Sa2VA是一种近期的语言引导图像与视频密集定位模型，在多个分割基准上达到当前最优表现，广受关注。然而我们发现，其在参考视频目标分割任务中未达预期效果。分析表明，训练与推理过程存在不一致是主要瓶颈。为此，我们提出改进版本Sa2VA-i，修复该问题并显著提升性能。实际结果显示，Sa2VA-i在多个视频基准上刷新了最佳纪录：在MeViS上提升+11.6 J&F，Ref-YT-VOS上+1.4，Ref-DAVIS上+3.3，ReVOS上+4.1，均使用相同原版Sa2VA检查点。更进一步，Sa2VA-i-1B模型在MeViS上表现与原始的Sa2VA-26B相当。本工作揭示了看似微小实现细节的重要性，为参考视频分割领域提供关键洞见。代码与更新模型已开源。
原文摘要 · Abstract (English)
Sa2VA is a recent model for language-guided dense grounding in images and video that achieves state-of-the-art results on multiple segmentation benchmarks and that has become widely popular. However, we found that Sa2VA does not perform according to its full potential for referring video object segmentation tasks. We identify inconsistencies between training and inference procedures as the key factor holding it back. To mitigate this issue, we propose an improved version of Sa2VA, Sa2VA-i, that rectifies these issues and improves the results. In fact, Sa2VA-i sets a new state of the art for multiple video benchmarks and achieves improvements of up to +11.6 J&F on MeViS, +1.4 on Ref-YT-VOS, +3.3 on Ref-DAVIS and +4.1 on ReVOS using the same Sa2VA checkpoints. With our fixes, the Sa2VA-i-1B model even performs on par with the original Sa2VA-26B model on the MeViS benchmark. We hope that this work will show the importance of seemingly trivial implementation details and that it will provide valuable insights for the referring video segmentation field. We provide the code and updated models at https://github.com/kumuji/sa2va-i
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。