解决视频语义分割在跨域场景下的闪烁问题,提升模型泛化能力。
Time2General: Learning Spatiotemporal Invariant Representations for Domain-Generalization Video Semantic Segmentation
- 用时空记忆解码器聚合多帧上下文,实现无需显式匹配的时序一致预测。
- 在多个驾驶数据集上跨域准确率显著提升,且推理速度达18 FPS。
- 适合需要高稳定性视频分割的自动驾驶系统部署。
领域泛化的视频语义分割(DGVSS)在单一标注驾驶域上训练后,可直接部署于未见域,无需目标标签或测试时自适应,同时保持视频流中时序一致的预测结果。实际中,领域偏移和时序采样偏移破坏基于对应关系的传播与固定步长的时序聚合,导致即使在标签稳定的区域也出现严重帧间闪烁。本文提出Time2General,一个基于稳定查询的DGVSS框架。它引入时空记忆解码器,将多帧上下文聚合为片段级时空记忆,并解码出无显式对应传播的时序一致帧级掩码。为进一步抑制闪烁并增强对不同采样率的鲁棒性,提出掩码化时序一致性损失,正则化不同步长下的预测差异,并随机化训练步长以暴露模型于多样时序间隙。在多个驾驶基准上的大量实验表明,Time2General在跨域准确率和时序稳定性方面均显著优于现有DGSS和VSS基线,同时运行速度高达18 FPS。代码将在评审结束后发布。
原文摘要 · Abstract (English)
Domain Generalized Video Semantic Segmentation (DGVSS) is trained on a single labeled driving domain and is directly deployed on unseen domains without target labels and test-time adaptation while maintaining temporally consistent predictions over video streams. In practice, both domain shift and temporal-sampling shift break correspondence-based propagation and fixed-stride temporal aggregation, causing severe frame-to-frame flicker even in label-stable regions. We propose Time2General, a DGVSS framework built on Stability Queries. Time2General introduces a Spatio-Temporal Memory Decoder that aggregates multi-frame context into a clip-level spatio-temporal memory and decodes temporally consistent per-frame masks without explicit correspondence propagation. To further suppress flicker and improve robustness to varying sampling rates, the Masked Temporal Consistency Loss is proposed to regularize temporal prediction discrepancies across different strides, and randomize training strides to expose the model to diverse temporal gaps. Extensive experiments on multiple driving benchmarks show that Time2General achieves a substantial improvement in cross-domain accuracy and temporal stability over prior DGSS and VSS baselines while running at up to 18 FPS. Code will be released after the review process.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。