通过时空感知压缩,让视频推理分割更快更准。
STAC: Selective Spatiotemporal Aggregation and Compression for Video Reasoning Segmentation

- 用状态空间模型实现线性时序建模,动态判断冗余特征
- 压缩后保留85%关键帧,速度提升1.8倍且精度更高
- 适合需要实时处理长视频的视觉语言任务
视频推理分割需在复杂自然语言查询下对数百帧图像进行像素级目标追踪,生成密集的时空标记,其二次自注意力开销使长视频处理难以实现。现有方法虽采用标记压缩,但通常基于无时序上下文的编码器特征,导致选择时机过早,无法可靠评估内容冗余。真正有效的压缩需具备上下文感知能力,但全分辨率获取上下文仍存在与压缩目标相同的二次开销。状态空间模型可解决此矛盾:其线性递推机制以 $/mathcal{O}(T)$ 成本实现每个标记对时序上下文的选择性依赖,生成可评估冗余性的表示。基于此,本文提出选择性时空聚合与压缩(STAC),通过解耦的双向空间扫描和因果时序扫描丰富特征,利用递推生成的冗余信息实现分层压缩,并以分割目标优化自适应阈值。STAC 实现 85% 的标记减少和 1.8× 的加速,在零样本流式设置下超越无压缩基线模型,在推理分割基准上表现更优。代码已开源。
原文摘要 · Abstract (English)
Video reasoning segmentation demands pixel-accurate object tracking across hundreds of frames under complex natural language queries, producing dense spatiotemporal tokens whose quadratic self-attention cost makes long-video processing prohibitive. Existing methods address this through token compression, yet typically operate on encoder features lacking temporal context, constraining selection before content redundancy can be reliably assessed. Informed compression requires contextual awareness, but acquiring that awareness at full resolution incurs the same quadratic cost compression aims to reduce. State-space models resolve this constraint, as their linear recurrence selectively conditions each token on temporal context at $\mathcal{O}(T)$ cost, producing representations where content redundancy becomes assessable. Building on this, Selective SpatioTemporal Aggregation and Compression (STAC) enriches features via decoupled bidirectional spatial and causal temporal scanning, leveraging recurrence-derived redundancy for hierarchical compression with adaptive thresholds optimised with segmentation objective. STAC achieves 85% token reduction and 1.8$\times$ speedup while surpassing compression-free baselines on reasoning segmentation benchmarks in a zero-shot streaming-compatible setting. Code is available \href{https://github.com/MCG-NKU/nku-video}{here}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。