提出DETRIS框架，用密集连接提升跨模态特征对齐，适合大模型微调。
Densely Connected Parameter-Efficient Tuning for Referring Image Segmentation
- 通过层间密集连接增强视觉特征传播，促进跨模态交互
- 在挑战性基准上性能超越现有方法0.9%~1.8%，仅更新少量参数
- 特别适用于视觉与语言编码器不匹配的场景，适合大模型高效微调
在计算机视觉领域，参数高效微调（PET）正逐步取代传统预训练+全量微调范式。尽管其在大基础模型中表现优异，但现有方法多针对单模态优化。少数探索尝试了对齐编码器（如CLIP），却未深入研究非对齐编码器场景。这些方法在非对齐编码器下表现不佳，因无法有效对齐多模态特征。本文提出DETRIS，一种通过每层与所有前序层建立密集连接，以增强低秩视觉特征传播的参数高效微调框架，实现有效的跨模态特征交互，并引入文本适配器优化文本特征。该方法在多个挑战性基准上显著优于当前最优方法，仅需0.9%至1.8%的主干参数更新。项目代码已开源。
原文摘要 · Abstract (English)
In the domain of computer vision, Parameter-Efficient Tuning (PET) is increasingly replacing the traditional paradigm of pre-training followed by full fine-tuning. PET is particularly favored for its effectiveness in large foundation models, as it streamlines transfer learning costs and optimizes hardware utilization. However, the current PET methods are mainly designed for single-modal optimization. While some pioneering studies have undertaken preliminary explorations, they still remain at the level of aligned encoders (e.g., CLIP) and lack exploration of misaligned encoders. These methods show sub-optimal performance with misaligned encoders, as they fail to effectively align the multimodal features during fine-tuning. In this paper, we introduce DETRIS, a parameter-efficient tuning framework designed to enhance low-rank visual feature propagation by establishing dense interconnections between each layer and all preceding layers, which enables effective cross-modal feature interaction and adaptation to misaligned encoders. We also suggest using text adapters to improve textual features. Our simple yet efficient approach greatly surpasses state-of-the-art methods with 0.9% to 1.8% backbone parameter updates, evaluated on challenging benchmarks. Our project is available at \url{https://github.com/jiaqihuang01/DETRIS}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。