提出诊断框架，拆解视频实例分割中的性能瓶颈。
Mind the Gap: Disentangling Performance Bottlenecks in Video Instance Segmentation
- 将身份与分类分配建模为整数线性规划，实现无模型诊断
- 发现在线方法在遮挡下跟踪误差超20 AP，随视频变长急剧恶化
- 开发可视化工具TrackLens，定位失败模式，适合算法优化者
在视频实例分割（VIS）中，分类、分割和跟踪目标联合评估，但各自对性能损失的贡献不明确。本文提出一种诊断框架，将身份与类别分配建模为整数线性规划（ILP），构建一个无模型的基准代理，分层隔离各类错误来源。应用于七种覆盖在线与离线范式的VIS方法，涵盖YouTube-VIS 2019/2021及OVIS诊断子集，分析揭示一致结论：在线方法在重遮挡下跟踪不稳定，性能差距超过20 AP，且随视频长度和实例密度增加而显著上升。尽管语义分类在标准基准上贡献明显，但在跟踪失效最严重时影响可忽略。更强主干网络虽能提升基础分数，但对跟踪AP差距影响甚微，证实时间脆弱性属算法而非表征问题。为补充该基准，引入TrackLens可视化工具，将差距大小映射为可观察的查询级失败模式。二者共同为解决VIS核心挑战——长期稳定时序关联，提供系统性基础。
原文摘要 · Abstract (English)
In Video Instance Segmentation (VIS), classification, segmentation, and tracking objectives are jointly evaluated, but their individual contributions to performance loss remain opaque. We introduce a diagnostic framework that formulates identity and class assignment as an Integer Linear Program (ILP), yielding a model-agnostic oracle that hierarchically isolates each error source. Applied to seven VIS methods spanning online and offline paradigms across YouTube-VIS 2019/2021 and a diagnostic subset of OVIS, our analysis reveals a consistent picture. Tracking instability is a critical bottleneck for online methods, with gaps exceeding 20 AP under heavy occlusion, and grows sharply with video length and instance density. While semantic classification contributes meaningfully on standard benchmarks, its impact becomes negligible where tracking fails most. Although stronger backbones substantially lift default scores, they leave AP tracking gaps largely intact, confirming that temporal fragility is algorithmic rather than purely representational. To complement the oracle, we introduce TrackLens, a visual tool that translates gap magnitude into observable, query-level failure modes. Together, these tools provide a systematic foundation for targeting VIS's core challenge: robust long-term temporal association.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。