用时空注意力提升自动驾驶视频语义分割的稳定性和精度
Spatio-Temporal Attention for Consistent Video Semantic Segmentation in Automated Driving
- 引入时空注意力机制，让模型同时理解空间与时间信息
- 在Cityscapes和BDD100k上实现9.2%一致性提升，mIoU最高增1.76%
- 兼容多种模型架构，轻量与大模型均有效，适合实际部署
深度神经网络，尤其是基于Transformer的架构，在环境感知的语义分割中取得了显著进展。然而，现有模型独立处理每一帧视频，未能利用时间一致性，这会严重影响动态场景下的准确性和稳定性。本文提出一种时空注意力（STA）机制，将Transformer注意力模块扩展为支持多帧上下文，从而生成鲁棒的时序特征表示。该方法修改标准自注意力，以处理时空特征序列，保持计算效率并仅需对现有架构做最小改动。STA在多种Transformer架构上表现良好，适用于轻量级与大规模模型。在Cityscapes和BDD100k数据集上的全面评估显示，相比单帧基线，其时序一致性指标提升9.20个百分点，平均交并比（mIoU）最高提升1.76个百分点。结果表明，STA是视频语义分割应用中有效的架构增强方案。
原文摘要 · Abstract (English)
Deep neural networks, especially transformer-based architectures, have achieved remarkable success in semantic segmentation for environmental perception. However, existing models process video frames independently, thus failing to leverage temporal consistency, which could significantly improve both accuracy and stability in dynamic scenes. In this work, we propose a Spatio-Temporal Attention (STA) mechanism that extends transformer attention blocks to incorporate multi-frame context, enabling robust temporal feature representations for video semantic segmentation. Our approach modifies standard self-attention to process spatio-temporal feature sequences while maintaining computational efficiency and requiring minimal changes to existing architectures. STA demonstrates broad applicability across diverse transformer architectures and remains effective across both lightweight and larger-scale models. A comprehensive evaluation on the Cityscapes and BDD100k datasets shows substantial improvements of 9.20 percentage points in temporal consistency metrics and up to 1.76 percentage points in mean intersection over union compared to single-frame baselines. These results demonstrate STA as an effective architectural enhancement for video-based semantic segmentation applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。