提出TROY-VIS模型，实现高效实时的开放词汇视频实例分割。
Towards Real-Time Open-Vocabulary Video Instance Segmentation
- 分离注意力增强模块加速多模态信息交互
- 20倍提速至25帧/秒，精度相当或更优
- 适合移动机器人、AR等实时场景应用
本文针对实时开放词汇视频实例分割（OV-VIS）的挑战，分析了当前主流基础模型的计算瓶颈，提出新方法TROY-VIS，显著提升处理速度并保持高精度。引入三项关键技术：(1) 解耦注意力特征增强器，加快不同模态与尺度间的信息交互；(2) 快速嵌入记忆（Flash Embedding Memory），实现物体类别文本嵌入的高速获取；(3) 核心插值法，利用视频的时间连续性。实验表明，TROY-VIS在两个大规模基准数据集BURST和LV-VIS上达到最佳精度-速度权衡，运行速度达25 FPS，较GLEE-Lite快20倍（1.25 FPS），精度相当甚至更优。结果证明其在移动机器人、增强现实等动态环境中的实时应用潜力。代码与模型将开源于https://github.com/google-research/troyvis。
原文摘要 · Abstract (English)
In this paper, we address the challenge of performing open-vocabulary video instance segmentation (OV-VIS) in real-time. We analyze the computational bottlenecks of state-of-the-art foundation models that performs OV-VIS, and propose a new method, TROY-VIS, that significantly improves processing speed while maintaining high accuracy. We introduce three key techniques: (1) Decoupled Attention Feature Enhancer to speed up information interaction between different modalities and scales; (2) Flash Embedding Memory for obtaining fast text embeddings of object categories; and, (3) Kernel Interpolation for exploiting the temporal continuity in videos. Our experiments demonstrate that TROY-VIS achieves the best trade-off between accuracy and speed on two large-scale OV-VIS benchmarks, BURST and LV-VIS, running 20x faster than GLEE-Lite (25 FPS v.s. 1.25 FPS) with comparable or even better accuracy. These results demonstrate TROY-VIS's potential for real-time applications in dynamic environments such as mobile robotics and augmented reality. Code and model will be released at https://github.com/google-research/troyvis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。