用表征引导微调，让大模型在机器人视觉中更高效精准
RepSAM: Bridging Foundation Models to Robotic Vision via Representation-Guided Adaptation
- 基于表征差异动态分配参数，实现高效微调
- 仅用4小时训练即达全量微调97.9%性能，参数减少158倍
- 特别适合透明物、杂乱场景等机器人视觉难题
尽管像SAM这样的基础模型具备零样本能力，但其在非结构化环境中的机器人感知仍面临挑战。本文发现性能下降源于变压器层间表征分布不均：浅层存在显著领域差距（CKA < 0.5），深层则能有效迁移（CKA > 0.7）。基于此，提出RepSAM——一种表征引导的参数高效微调框架。该方法结合理论支持的CKA引导秩分配策略与多模态融合模块，可稳健处理透明物体、杂乱场景等复杂机器人场景。在六个基准和机器人操作任务上评估显示，RepSAM达到全量微调97.9%的性能（mIoU 89.0% vs. 90.9%），可训练参数减少158倍（632M → 4.0M）。相比DoRA提升7.9% mIoU，仅需单张A100 GPU训练4小时（全量微调需384 GPU小时，减少96倍）。结果统计显著（p < 0.01），机器人操作成功率绝对提升12.0%（相较LoRA RGB基线）。
原文摘要 · Abstract (English)
Robotic perception in unstructured environments remains challenging despite the zero-shot capabilities of foundation models such as SAM. This work attributes performance degradation to non-uniform representation shifts across transformer layers: shallow layers exhibit substantial domain gaps (CKA < 0.5), whereas deep layers transfer effectively (CKA > 0.7). Based on this observation, we propose RepSAM, a representation-guided parameter-efficient fine-tuning (PEFT) framework for adapting foundation models to robotic vision. RepSAM employs a theoretically grounded CKA-guided rank allocation strategy combined with a multi-modal fusion module for robust handling of challenging robotic scenarios, including transparent objects and cluttered scenes. Experimental evaluation across six benchmarks and robotic manipulation tasks demonstrates that RepSAM achieves 97.9% of full fine-tuning performance (89.0% vs. 90.9% mIoU) while reducing trainable parameters by 158x (from 632M to 4.0M). RepSAM outperforms DoRA by 7.9% mIoU with just 4 hours of training on a single A100 GPU (a 96x reduction from full fine-tuning, which takes 384 GPU-hours). These improvements are statistically significant (p < 0.01) and translate to a 12.0% absolute improvement in robotic manipulation success rates over the LoRA (RGB) baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。