用动态局部先验实现高效微调,提升复杂目标分割精度
Controllable-LPMoE: Adapting to Challenging Object Segmentation via Dynamic Local Priors from Mixture-of-Experts
- 通过专家混合机制动态生成局部先验,减少可训练参数
- 在多个二值分割任务上超越31种先进方法,性能显著提升
- 适合需要高效微调且对精度要求高的视觉分割场景
大规模基础模型为下游目标分割任务提供强大特征表示。然而,通过全参数微调适配特定任务时,大量参数更新带来显著计算开销,成为训练效率的瓶颈。现有方法尝试通过嵌入可训练提示来微调冻结模型,但这些提示缺乏内在语义先验,限制了模型适应性。本文提出一种新型基于动态先验的微调范式Controllable-LPMoE,仅需少量可训练参数,通过动态调控局部先验来增强基础模型对细粒度感知的能力。具体而言,构建轻量级动态混合局部先验提取器,利用异质卷积从输入图像中捕捉多样化局部先验,并通过门控网络动态输出后续微调所需的专家先验。此外,设计双向交互适配器,结合余弦对齐可变形注意力与通道导向自适应缩放,实现冻结特征与可训练特征间的高效交互与重构。大量实验验证了Controllable-LPMoE的优势,在多个二值对象分割任务上表现优于31种前沿方法,展现出卓越的分割性能和多任务适应能力。
原文摘要 · Abstract (English)
Large-scale foundation models provide powerful feature representations for downstream object segmentation tasks. However, when adapted to specific tasks through the full-parameter fine-tuning, the enormous parameters being updated often results in significant computational overhead, creating a bottleneck in training efficiency. Although existing methods attempt to fine-tune frozen models by directly embedding trainable prompts, these prompts lack inherent semantic priors, limiting the adaptability of large-scale models. In this paper, we propose a novel dynamic priors-based fine-tuning paradigm with fewer trainable parameters, dubbed Controllable-LPMoE, which adaptively modulates frozen foundation models by dynamically controlling local priors to enhance fine-grained perception for specific segmentation tasks. More specifically, we construct a lightweight dynamic mixed local priors extractor that captures diverse local priors from input images through heterogeneous convolutions while employing a gating network to dynamically output expert priors required for the subsequent fine-tuning. Furthermore, we design a bi-directional interaction adapter that employs cosine-aligned deformable attention and channel-oriented adaptive scale enhancement to interact and restructure between frozen and trainable features, achieving efficient fine-tuning. Extensive experiments validate the superiority of our \href{https://github.com/CSYSI/Controllable-LPMoE} {Controllable-LPMoE} approach, demonstrating excellent segmentation performance compared to 31 state-of-the-art (SOTA) methods and adaptability to multiple binary object segmentation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。