让视觉语言模型的意图变成机器人精准动作，还能跨任务复用。
GAE: Unleashing Physical Potential of VLM with Generalizable Action Expert
- 用稀疏3D关键点表达高层意图，再由GAE转为连续动作轨迹。
- 在15万条仿真与真实机器人轨迹上预训练，支持多场景泛化。
- 只需微调视觉语言模型，即可快速适配新任务，效率高。
视觉语言模型具备强大的推理与规划能力，但将其预测结果转化为精确机器人动作仍是核心挑战。现有视觉-语言-动作方法通常将推理与动作生成耦合，导致泛化能力有限。本文提出通用动作专家（GAE），一种任务无关模型，可将稀疏几何规划转换为密集机器人动作。该方法引入稀疏几何接口：视觉语言模型预测代表高层意图的稀疏3D关键点，GAE则结合实时点云观测，将这些关键点映射为连续动作轨迹。GAE在包含15万条轨迹的大规模点云-轨迹数据集上进行预训练，涵盖仿真与真实机器人数据。为进一步提升效率与泛化性，提出动作预训练-点云微调（APPF）策略，解耦动作动力学学习与几何定位。预训练后GAE固定不动，可复用于下游任务，仅需对视觉语言模型进行轻量级微调以生成稀疏接口。实验表明，该方法在多样视觉域、相机视角和自然语言指令下均表现优异且具有良好泛化能力。
原文摘要 · Abstract (English)
Vision-language models demonstrate strong reasoning and planning abilities, yet grounding these predictions into precise robot actions remains a central challenge. Existing Vision-Language-Action methods typically entangle reasoning and action generation, leading to limited generalization. We propose Generalizable Action Expert (GAE), a task-agnostic model that converts sparse geometric plans into dense robot actions. Our approach introduces a sparse geometric interface: the VLM predicts sparse 3D waypoints representing high-level intention, while GAE maps these waypoints together with real-time point cloud observations to continuous action trajectories. GAE is pretrained on a large-scale pointcloud-trajectory dataset comprising 150k trajectories from both simulation and real-world robots. To further improve efficiency and generalization, we introduce an Action Pre-training, Pointcloud Fine-tuning (APPF) scheme that decouples learning action dynamics from geometry grounding. After pretraining, GAE is frozen and reused across downstream tasks, requiring only lightweight fine-tuning of the VLM to produce the sparse interface. Experiments show that our method achieves strong performance and generalization across diverse visual domains, camera viewpoints, and natural language instructions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。