用线性匹配降低内存开销，实现长视频高分辨率分割
LiVOS: Light Video Object Segmentation with Gated Linear Matching
- 采用线性注意力重构记忆匹配，将复杂度从二次降至常数
- 在MOSE和DAVIS上分别达到64.8和85.1 J&F，超越非STM方法
- 仅需STM一半内存即可处理4096p视频，适合消费级硬件
半监督视频对象分割（VOS）主要依赖时空记忆（STM）网络，通过softmax注意力在时空中存储历史特征以分割当前帧。然而，由于softmax匹配具有二次复杂度，导致内存受限，难以应对更长视频和更高分辨率。为此，我们提出LiVOS，一种轻量级记忆网络，采用线性注意力实现线性匹配，将记忆匹配重构为递归过程，将二次注意力矩阵降为固定大小、与时空无关的2D状态。为进一步提升选择性，引入门控线性匹配，通过数据相关的门矩阵乘以状态矩阵，控制信息保留或丢弃。在多个基准测试中验证了该方法的有效性：在MOSE上取得64.8 J&F，DAVIS上达85.1 J&F，超越所有非STM方法，并缩小与STM方法的差距。对于更长、更高分辨率视频，其内存消耗仅为STM方法的53%，可在32G消费级显卡上支持4096p推理——这一能力此前成本过高，为长视频高分辨率基础模型铺平道路。
原文摘要 · Abstract (English)
Semi-supervised video object segmentation (VOS) has been largely driven by space-time memory (STM) networks, which store past frame features in a spatiotemporal memory to segment the current frame via softmax attention. However, STM networks face memory limitations due to the quadratic complexity of softmax matching, restricting their applicability as video length and resolution increase. To address this, we propose LiVOS, a lightweight memory network that employs linear matching via linear attention, reformulating memory matching into a recurrent process that reduces the quadratic attention matrix to a constant-size, spatiotemporal-agnostic 2D state. To enhance selectivity, we introduce gated linear matching, where a data-dependent gate matrix is multiplied with the state matrix to control what information to retain or discard. Experiments on diverse benchmarks demonstrated the effectiveness of our method. It achieved 64.8 J&F on MOSE and 85.1 J&F on DAVIS, surpassing all non-STM methods and narrowing the gap with STM-based approaches. For longer and higher-resolution videos, it matched STM-based methods with 53% less GPU memory and supports 4096p inference on a 32G consumer-grade GPU--a previously cost-prohibitive capability--opening the door for long and high-resolution video foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。