将视频中精细动作段落分割并转化为机器人可执行指令，提升人机协作安全性。
RoboSubtaskNet: Temporal Sub-task Segmentation for Human-to-Robot Skill Transfer in Real-World Environments
- 用改进的MS-TCN结合注意力增强特征，捕捉短时动作过渡
- 在真实数据集上达到94.2%的子任务分割准确率，物理实验成功率达91.25%
- 专为机器人操作设计的数据集与端到端控制管道，适合实际部署
在长而未剪辑的视频中精确识别和分类细粒度子任务段，对安全的人机协作至关重要。不同于通用行为识别，协作操作需要直接可被机器人执行的子任务标签。本文提出RoboSubtaskNet，一种多阶段人到机器人的子任务分割框架，融合增强注意力的I3D特征（RGB+光流）与采用斐波那契膨胀策略的改进型MS-TCN，以更好捕捉如抓取-放置等短时过渡。网络通过组合交叉熵与时间正则化（截断MSE和过渡感知项）的复合目标函数训练，减少过分割并促进有效子任务序列。为弥合视觉基准与控制之间的差距，我们引入RoboSubtask数据集，涵盖医疗与工业演示，标注至子任务层级，并支持向机械臂基础操作的确定性映射。实验证明，RoboSubtaskNet在GTEA和自建的RoboSubtask基准上优于MS-TCN和MS-TCN++（边界敏感与序列指标），在长时早餐数据集上仍具竞争力。具体表现：在GTEA上取得F1@50=79.5%，Edit=88.6%，Acc=78.9%；在Breakfast上为F1@50=30.4%，Edit=52.0%，Acc=53.5%；在RoboSubtask上达F1@50=94.2%，Edit=95.6%，Acc=92.2%。进一步在7-DoF Kinova Gen3机械臂上验证端到端感知-执行流程，物理实验整体任务成功率约91.25%。结果表明，该方法实现了从子任务级视频理解到真实场景机器人操作的可行路径。
原文摘要 · Abstract (English)
Temporally locating and classifying fine-grained sub-task segments in long, untrimmed videos is crucial to safe human-robot collaboration. Unlike generic activity recognition, collaborative manipulation requires sub-task labels that are directly robot-executable. We present RoboSubtaskNet, a multi-stage human-to-robot sub-task segmentation framework that couples attention-enhanced I3D features (RGB plus optical flow) with a modified MS-TCN employing a Fibonacci dilation schedule to capture better short-horizon transitions such as reach-pick-place. The network is trained with a composite objective comprising cross-entropy and temporal regularizers (truncated MSE and a transition-aware term) to reduce over-segmentation and to encourage valid sub-task progressions. To close the gap between vision benchmarks and control, we introduce RoboSubtask, a dataset of healthcare and industrial demonstrations annotated at the sub-task level and designed for deterministic mapping to manipulator primitives. Empirically, RoboSubtaskNet outperforms MS-TCN and MS-TCN++ on GTEA and our RoboSubtask benchmark (boundary-sensitive and sequence metrics), while remaining competitive on the long-horizon Breakfast benchmark. Specifically, RoboSubtaskNet attains F1 @ 50 = 79.5%, Edit = 88.6%, Acc = 78.9% on GTEA; F1 @ 50 = 30.4%, Edit = 52.0%, Acc = 53.5% on Breakfast; and F1 @ 50 = 94.2%, Edit = 95.6%, Acc = 92.2% on RoboSubtask. We further validate the full perception-to-execution pipeline on a 7-DoF Kinova Gen3 manipulator, achieving reliable end-to-end behavior in physical trials (overall task success approx 91.25%). These results demonstrate a practical path from sub-task level video understanding to deployed robotic manipulation in real-world settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。