用3D高斯表示视觉，让机器人更精准理解空间动作。
GST-VLA: Structured Gaussian Spatial Tokens for 3D Depth-Aware Vision-Language-Action Models
- 将图像转为128个带几何属性的3D高斯点，保留深度与方向信息
- 在三个数据集上动作成功率超现有模型2%-5.4%，最高达96.4%
- 适合需要精确空间推理的机器人任务，如抓取与路径规划
VLA模型将视觉输入编码为无几何结构的2D图像块。本文提出GST-VLA，引入两种创新：首先，高斯空间分词器（GST）将冻结的稠密深度与语义特征转换为128个各向异性3D高斯原型，每个由度量残差均值μ∈ℝ³、对数尺度协方差logσ∈ℝ³和可学习不透明度α∈(0,1)参数化，其协方差特征结构编码局部表面朝向，不透明度提供几何置信度，这是标量深度无法获取的。空间注意力池化通过学习查询，将固定令牌预算集中于几何显著区域。其次，3D深度感知思维链（DA-CoT）监督四个结构化中间空间思考：3D物体定位、抓取接触几何、成对距离及粗略SE(3)路径点，并作为训练损失中的显式生成目标。在每个VLM Transformer块中加入交叉注意力子层，使模型能直接访问原始256个高斯原型场。一个300M参数的流匹配动作专家采用混合专家前馈结构，通过条件ODE积分解码7自由度动作增量，条件输入包括VLM隐藏状态与DA-CoT输出，经双交叉注意力实现。模型在三个渐进阶段联合优化组合损失ℒ_flow + ℒ_CoT + ℒ_depth，最终在LIBERO上达到96.4%（+2.0%），SimplerEnv上达到80.2%（+5.4%）。消融实验验证了各组件、每项思维链及训练阶段的独立与协同增益，且优势集中在高精度需求任务上。
原文摘要 · Abstract (English)
VLA models encode visual observations as 2D patch tokens with no intrinsic geometric structure. We introduce GST-VLA with two contributions. First, the Gaussian Spatial Tokenizer (GST) converts frozen dense depth and frozen semantic patch features into $N_g{=}128$ anisotropic 3D Gaussian primitives, each parameterized by a metric residual mean $μ\in \mathbb{R}^3$, log-scale covariance $\log σ\in \mathbb{R}^3$, and learned opacity $α\in (0,1)$. The covariance eigenstructure encodes local surface orientation, and opacity provides per-primitive geometric confidence, both inaccessible from scalar depth. Spatial attention pooling with learned queries concentrates the fixed token budget on geometrically salient regions rather than distributing uniformly. Second, 3D Depth-Aware Chain-of-Thought (DA-CoT) reasoning supervises four structured intermediate spatial thoughts, covering 3D object grounding, grasp affordance contact geometry, pairwise metric distances, and coarse SE(3) waypoints, as explicit generation targets in the training loss. A cross-attention sublayer at every VLM transformer block provides direct access to the raw 256-primitive Gaussian field during DA-CoT generation. A 300M-parameter flow-matching action expert with mixture-of-experts feedforward sublayers decodes 7-DoF delta action chunks via conditional ODE integration, conditioned on both VLM hidden states and DA-CoT outputs through dual cross-attention. Trained with composite $\mathcal{L}_\mathrm{flow} + \mathcal{L}_\mathrm{CoT} + \mathcal{L}_\mathrm{depth}$ across three progressive stages, GST-VLA achieves 96.4% on LIBERO (+2.0%), and 80.2% on SimplerEnv (+5.4%). Ablations isolate the contribution of each GST component, each DA-CoT thought, and each training stage, confirming independent and synergistic gains concentrated on precision demanding tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。