视觉+时序逻辑直接生成机器人动作轨迹,无需物体几何信息
Vision-TL-Action: Neuro-Symbolic Trajectory Generation from Visual Observations and Temporal Logic

- 用多视角图像和时序逻辑图融合生成动作序列
- 在Panda任务上达67.45%成功率(1024次采样),优于基线
- 适合需要视觉感知与复杂任务规划的机器人系统
时序逻辑(TL)为长周期机器人任务提供了组合式表述语言,但现有TL条件轨迹生成器可通过在任务图中编码精确物体几何来规避感知到符号的绑定。我们提出Vision-TL-Action,从多视角图像、无坐标系的TL语法图和机器人初始状态生成动作轨迹。通过双向交叉注意力融合TL节点标记与空间视觉标记,生成的表示用于条件流匹配轨迹生成器。视觉标记仅附加归一化图像平面位置和相机视图标识符,训练阶段引入谓词到区域的目标以促进对参考物体的语义定位。遵循该领域惯例,使用Success@K评估模型表现,即在K次采样轨迹中至少有一条满足TL规范的比例。在Panda任务上,模型达到67.45% Success@1024,优于60.11%的基于真值状态的基线;在AntMaze任务上达96.35% Success@256,接近真值基线96.88%。分辨率与干预实验表明,空间细节依赖于语义定位,谓词身份同时影响注意力分布与性能。结果证明了从视觉观测和结构化TL目标到动作轨迹的直接映射,无需推理时提供物体几何信息。代码已开源。
原文摘要 · Abstract (English)
Temporal logic (TL) provides a compositional language for the formulation of long horizon robotic tasks, but existing TL-conditioned trajectory generators can sidestep perception-to-symbol binding by encoding exact object geometry in the task graph. We introduce \emph{Vision-TL-Action}, which generates action trajectories from multi-view images, a coordinate-free TL syntax graph, and the robot initial state. TL-node tokens and spatial visual tokens are fused through bidirectional cross-attention, and the resulting representation conditions a flow-matching trajectory generator. Visual tokens are augmented only with normalized image-plane locations and camera-view identifiers, while a training-only predicate-to-region objective encourages grounding to referenced objects. Consistent with prior work in this domain, we evaluate the model using Success@$K$, the fraction of tasks for which at least one of K sampled trajectories satisfies the TL specification. On Panda task, our model achieves 67.45% Success@1024, compared with 59.11% for the oracle-state baseline. On AntMaze task, it achieves 96.35% Success@256, comparable to the oracle result of 96.88%. Resolution and intervention studies show that spatial detail depends on semantic grounding and predicate identity affects both attention and performance. These results demonstrate a direct mapping from visual observations and structured TL goals to action trajectories without requiring object geometry at inference. Code is available at https://github.com/AricLau07/vision-tl-action.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。