轻量化音频视觉分割模型，速度与精度兼得。
LightAVSeg: Lightweight Audio-Visual Segmentation
- 用解耦设计替代密集跨模态注意力，计算成本线性增长
- 20.5M参数下达50.4 mIoU，是轻量级方法新标杆
- 新增对齐损失无推理开销，适合移动端部署
音频-视觉分割（AVS）旨在视频中精确定位发声物体的像素级位置。现有模型依赖密集跨模态注意力，计算复杂度呈二次增长，限制了其在资源受限场景的部署。多数高效方法仅关注主干网络压缩，忽视交互模块这一主要瓶颈。本文提出LightAVSeg，通过解耦语义过滤与空间定位设计，将交互计算成本降至与空间分辨率线性相关。此外，引入辅助对齐损失以训练时强制语义一致性，且推理无额外开销。大量实验表明，LightAVSeg在轻量级方法中达到新基准：仅20.5M参数（约为AVSegFormer的1/7），在MS3基准上实现50.4 mIoU，并可在移动处理器上实现高效推理。
原文摘要 · Abstract (English)
Audio-Visual Segmentation (AVS) targets pixel level localization of sounding emitting objects in videos. However, existing models rely on dense cross-modal attention with quadratic computational cost, limiting their suitability for resource efficient deployment. Most efficiency oriented methods focus on backbone reduction and overlook the interaction module as the primary bottleneck. This paper proposes LightAVSeg, a lightweight framework that replaces heavy attention with a decoupled design for semantic filtering and spatial grounding, resulting in interaction costs that scale linearly with spatial resolution. Furthermore, we introduce an auxiliary alignment loss to enforce semantic consistency during training with zero inference overhead. Extensive experiments demonstrate that LightAVSeg achieves a new state-of-the-art among lightweight methods: with 20.5M parameters ~1/7 of AVSegFormer), it reaches 50.4 mIoU on the MS3 benchmark and enables efficient inference on a mobile processor.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。