针对指代图像分割中定位不准的问题,提出分阶段专家路由架构。
Spatio-Semantic Expert Routing Architecture with Mixture-of-Experts for Referring Image Segmentation
- 分两阶段引入表达式感知的轻量级专家精修模块。
- 在标准数据集上显著提升边界精度与空间连贯性,尤其对复杂表达效果明显。
- 仅更新1%参数即可稳定适配冻结主干网络,适合资源受限场景。
指代图像分割旨在为自然语言描述的图像区域生成像素级掩码。尽管预训练视觉-语言模型提升了语义定位能力,但现有方法仍依赖统一精修策略,难以匹配指代表达的多样化推理需求。因此,预测常出现碎片化区域、边界不准甚至错误对象,尤其在冻结预训练主干以节省计算成本时更为明显。为此,本文提出SERA——一种用于指代图像分割的时空语义专家路由架构。SERA在视觉-语言框架内设计两个互补阶段的轻量级、表达式感知专家精修机制。首先,提出SERA-Adapter,在选定主干块中插入表达式条件适配器,通过专家引导精修和跨模态注意力增强空间连贯性与边界精度;其次,引入SERA-Fusion,将中间视觉特征重塑为空间网格,并在多模态交互前应用几何保持型专家变换以强化表示。此外,设计轻量级路由机制自适应加权专家贡献,且兼容预训练表示。为确保在冻结编码器下路由稳定,SERA采用参数高效微调策略,仅更新归一化与偏置项,影响主干参数不足1%。在标准基准测试上的实验表明,SERA持续优于强基线,尤其在需精确空间定位与边界划分的表达上表现突出。
原文摘要 · Abstract (English)
Referring image segmentation aims to produce a pixel-level mask for the image region described by a natural-language expression. Although pretrained vision-language models have improved semantic grounding, many existing methods still rely on uniform refinement strategies that do not fully match the diverse reasoning requirements of referring expressions. Because of this mismatch, predictions often contain fragmented regions, inaccurate boundaries, or even the wrong object, especially when pretrained backbones are frozen for computational efficiency. To address these limitations, we propose SERA, a Spatio-Semantic Expert Routing Architecture for referring image segmentation. SERA introduces lightweight, expression-aware expert refinement at two complementary stages within a vision-language framework. First, we design SERA-Adapter, which inserts an expression-conditioned adapter into selected backbone blocks to improve spatial coherence and boundary precision through expert-guided refinement and cross-modal attention. We then introduce SERA-Fusion, which strengthens intermediate visual representations by reshaping token features into spatial grids and applying geometry-preserving expert transformations before multimodal interaction. In addition, a lightweight routing mechanism adaptively weights expert contributions while remaining compatible with pretrained representations. To make this routing stable under frozen encoders, SERA uses a parameter-efficient tuning strategy that updates only normalization and bias terms, affecting less than 1% of the backbone parameters. Experiments on standard referring image segmentation benchmarks show that SERA consistently outperforms strong baselines, with especially clear gains on expressions that require accurate spatial localization and precise boundary delineation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。