通过分治专家结构，让视觉语言模型更高效适配机器人控制。
VLA-GSE: Boosting Parameter-Efficient Fine-Tuning in VLA with Generalized and Specialized Experts
- 用谱分解分离通用与专用专家，提升参数效率
- 仅更新2.51%参数，零样本成功率达81.2%
- 适合需要知识保留的机器人控制任务
视觉-语言-动作（VLA）模型继承了预训练视觉-语言主干的丰富语义先验，但将其适配至机器人控制仍具挑战。全量微调（FFT）易在下游机器人数据上过拟合，并导致预训练视觉-语言能力灾难性遗忘。参数高效微调（PEFT）虽更好保留预训练知识，但现有方法仍难以有效适应机器人控制任务。为此，我们提出VLA-GSE（广义与专用专家），一种参数高效的VLA微调框架，在保持知识保留优势的同时提升控制适配能力。具体地，VLA-GSE通过谱分解冻结的主干网络，将主要奇异分量分配给通用专家（共享专家），其余不相交残差分量分配给专用专家（路由专家）。该分解在固定可训练参数预算下提升了适应能力。在相近参数预算下，VLA-GSE仅更新2.51%的模型参数，且持续优于强基线的全量微调与参数高效微调方法。其在LIBERO-Plus上实现81.2%平均零样本成功率，多模态理解基准上与LoRA相当保留预训练视觉语言模型能力，并在多种分布偏移下提升真实世界操作成功率。代码已开源：https://github.com/YuhuaJiang2002/VLA-GSE。
原文摘要 · Abstract (English)
Vision-language-action (VLA) models inherit rich visual-semantic priors from pre-trained vision-language backbones, but adapting them to robotic control remains challenging. Full fine-tuning (FFT) is prone to overfitting on downstream robotic data and catastrophic forgetting of pretrained vision-language capabilities. Parameter-efficient fine-tuning (PEFT) better preserves pre-trained knowledge, yet existing PEFT methods still struggle to adapt effectively to robot control tasks. To address this gap, we propose VLA-GSE, a parameter-efficient VLA fine-tuning framework that improves control adaptation while retaining PEFT's knowledge preservation advantage. Specifically, VLA-GSE (Generalized and Specialized Experts) is initialized by spectrally decomposing the frozen backbone, assigning leading singular components to generalized experts (shared experts) and disjoint residual components to specialized experts (routed experts). This decomposition improves adaptation capacity under a fixed trainable-parameter budget. Under a comparable parameter budget, VLA-GSE updates only 2.51% of the full model parameters and consistently outperforms strong FFT and PEFT baselines. It achieves 81.2% average zero-shot success on LIBERO-Plus, preserves pre-trained VLM capability comparably to LoRA on multimodal understanding benchmarks, and improves real-world manipulation success under multiple distribution shifts. Code is available at: https://github.com/YuhuaJiang2002/VLA-GSE
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。