用推理时概率平滑提升弱提示下SAM2的视频分割稳定性
Inference-Time Temporal Probability Smoothing for Stable Video Segmentation with SAM2 under Weak Prompts
- 基于光流和熵的不确定性估计，动态融合历史预测
- 在四组视频上实现边界一致性与对象持续性显著提升
- 无需重训练，适合实时交互式视频分割场景
交互式视频分割模型如SAM2在多种视觉领域表现出强泛化能力。然而，在弱用户监督下（例如仅在单帧提供稀疏点提示），其预测常出现时间不稳定性，包括边界闪烁、对象丢失和帧间对象范围不一致，限制了其在下游视频理解与控制任务中的可靠性。本文提出一种推理时的时间概率平滑方法，无需重新训练或修改架构即可提升基于SAM2的视频分割时间稳定性。该方法直接作用于每帧的分割概率图，结合基于光流的运动补偿、由分割熵导出的像素级不确定性估计以及前后向光流一致性，自适应地融合当前帧预测与运动对齐的历史估计，生成时间连贯的分割结果。我们在四组多样化的视频序列上使用一系列帧级与时间稳定性度量（包括运动补偿交并比、边界一致性、对象持续性、面积波动性）进行评估，实验结果表明，相比原始SAM2推理，该方法在时间稳定性上实现一致改进，同时保持空间精度。所提框架轻量、模型无关，适用于实时交互式视频分割。
原文摘要 · Abstract (English)
Interactive video segmentation models such as SAM2 have demonstrated strong generalization across diverse visual domains. However, under weak user supervision, for example, when sparse point prompts are provided on a single frame, their predictions often suffer from temporal instability, including flickering boundaries, object dropout, and inconsistent object extents across frames. These issues limit their reliability in downstream video understanding and control applications. In this paper, we propose an inference-time temporal probability smoothing method that improves the temporal stability of SAM2-based video segmentation without retraining or architectural modification. Our approach operates directly on per-frame segmentation probability maps and leverages optical-flow-based motion warping together with pixel-wise uncertainty estimates derived from segmentation entropy, and forward-backwards flow consistency. These signals are used to adaptively blend current-frame predictions with motion-aligned historical estimates, yielding temporally coherent segmentation outputs under weak prompts. We evaluate the proposed method on four diverse video sequences using a comprehensive set of frame-wise and temporal stability metrics, including motion-compensated IoU, boundary consistency, object persistence, and area volatility. Experimental results demonstrate consistent improvements in temporal stability over vanilla SAM2 inference while preserving spatial accuracy. The proposed framework is lightweight, model-agnostic, and well-suited for real-time, interactive video segmentation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。