针对视觉语言动作模型,提出自适应容量分配的微调方法
Adaptive Capacity Allocation for Vision Language Action Fine-tuning
- 用可学习路由器动态选择每层和输入的参数量,替代固定秩
- 在真实机器人上实现比标准LoRA高31.6%的多任务成功率
- 对秩的选择不敏感,适合跨任务、跨环境的物理智能应用
视觉语言动作模型(VLAs)在物理人工智能中应用日益广泛,但部署到未见过的环境、设备或任务仍需适配。参数高效微调(PEFT),尤其是LoRA,常用于VLA策略,但其暴露的容量控制参数——秩(rank)——并不通用:机器人任务所需内在秩高于语言微调,且随任务变化。小秩(如 $r \in \{4, 8\}$)足以适配大语言模型,而谱分析表明VLAs可能需要更大秩(如 $r \approx 128$)甚至接近全秩,多任务下这一差距更显著。本文提出LoRA-SP(Select-Prune),一种秩自适应微调方法,将固定秩更新替换为基于输入与层的容量分配。LoRA-SP采用类似SVD的参数化,使用小型路由器,其非负得分作为共享向量库中的奇异值。通过设定累积平方得分的能量目标 $E(k) \ge η$ 来确定活跃集合,直接关联逼近误差。训练中 $η$ 促使能量集中在少数方向,使路由器学会依赖更少向量,同时保持精度。该方法生成紧凑适配器,减少跨任务干扰,提升泛化能力。在未见过的AgileX PiPER机械臂上,四个真实机器人操作任务中,两种VLA骨干($π_0$ 和 SmolVLA)均显示,LoRA-SP以远少于可训练参数达到或超过全微调性能,且多任务成功率较标准LoRA最高提升31.6%,对秩选择仍具鲁棒性。
原文摘要 · Abstract (English)
Vision language action models (VLAs) are increasingly used for Physical AI, but deploying a pre-trained VLA model to unseen environments, embodiments, or tasks still requires adaptation. Parameter-efficient fine-tuning (PEFT), especially LoRA, is common for VLA policies, yet the exposed capacity knob, the rank, does not transfer uniformly: robotics transfer exhibits a higher and task-varying intrinsic rank than language fine-tuning. Small ranks suffice for LLMs (e.g., $r \in \{4, 8\}$), while spectral analyses indicate VLAs may require much larger ranks (e.g., $r \approx 128$) or near-full rank, a mismatch that worsens in multi-task settings. We present LoRA-SP (Select-Prune), a rank-adaptive fine-tuning method that replaces fixed-rank updates with input- and layer-wise capacity. LoRA-SP uses an SVD-style parameterization with a small router whose nonnegative scores act as singular values over a shared vector bank. The active set is chosen by an energy target on the cumulative squared scores $E(k) \ge η$, providing a direct link to approximation error via our spectral analysis. During training, $η$ concentrates energy on a few directions and teaches the router to rely on fewer vectors while preserving accuracy. This yields compact adapters that reduce cross-task interference and improve generalization. On four real-robot manipulation tasks collected on an unseen AgileX PiPER arm, across two VLA backbones ($π_0$ and SmolVLA), LoRA-SP matches or exceeds full fine-tuning with far fewer trainable parameters, and improves multi-task success by up to 31.6% over standard LoRA while remaining robust to rank choice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。