arXiv:2604.21053cs.ROcs.CV2026-04

让机器人更懂操作步骤,能推理下一步该做什么。

Neuro-Symbolic Manipulation Understanding with Enriched Semantic Event Chains

论文配图:Neuro-Symbolic Manipulation Understanding with Enriched Semantic Event Chains
图 1 · 摘自论文原文
  • 将语义事件链升级为带置信度的符号状态,支持实时决策。
  • 在三个数据集上提升下一步动作预测准确率,抗感知噪声更强。
  • 适合需要可解释性与鲁棒性的机器人操作系统使用。

在人类环境中运行的机器人系统需推理物体交互随时间的变化、当前执行的动作以及接下来可能的操作步骤。经典增强型语义事件链(eSECs)提供了可解释的操作关系描述,但主要为描述性,不支持不确定性感知的决策。本文提出eSEC-LAM,一种神经符号框架,将eSECs转化为显式的事件级符号状态以实现操作理解。该框架在经典eSECs基础上引入置信度感知谓词、功能物体角色、先验可操作性、原始动作抽象和显著性引导解释线索。这些丰富后的符号状态由基于基础模型的感知前端通过确定性谓词提取获得,当前动作推断和下一步原始动作预测则通过轻量级符号推理完成。我们在EPIC-KITCHENS-100、EPIC-KITCHENS VISOR和Assembly101上评估了该框架在动作识别、下一步原始动作预测、对感知噪声的鲁棒性及解释一致性方面的表现。实验结果表明,eSEC-LAM在动作识别上达到竞争水平,显著提升下一步原始动作预测性能,在感知条件退化时比传统符号方法和端到端视频基线更具鲁棒性,并提供基于明确关系证据的时间一致解释轨迹。这表明增强型语义事件链不仅能作为可解释的操作描述,还可作为神经符号动作推理的有效内部状态。

原文摘要 · Abstract (English)

Robotic systems operating in human environments must reason about how object interactions evolve over time, which actions are currently being performed, and what manipulation step is likely to follow. Classical enriched Semantic Event Chains (eSECs) provide an interpretable relational description of manipulation, but remain primarily descriptive and do not directly support uncertainty-aware decision making. In this paper, we propose eSEC-LAM, a neuro-symbolic framework that transforms eSECs into an explicit event-level symbolic state for manipulation understanding. The proposed formulation augments classical eSECs with confidence-aware predicates, functional object roles, affordance priors, primitive-level abstraction, and saliency-guided explanation cues. These enriched symbolic states are derived from a foundation-model-based perception front-end through deterministic predicate extraction, while current-action inference and next-primitive prediction are performed using lightweight symbolic reasoning over primitive pre- and post-conditions. We evaluate the proposed framework on EPIC-KITCHENS-100, EPIC-KITCHENS VISOR, and Assembly101 across action recognition, next-primitive prediction, robustness to perception noise, and explanation consistency. Experimental results show that eSEC-LAM achieves competitive action recognition, substantially improves next-primitive prediction, remains more robust under degraded perceptual conditions than both classical symbolic and end-to-end video baselines, and provides temporally consistent explanation traces grounded in explicit relational evidence. These findings demonstrate that enriched Semantic Event Chains can serve not only as interpretable descriptors of manipulation, but also as effective internal states for neuro-symbolic action reasoning.

机器人操作符号推理可解释性动作预测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。