将机器人长视频转为可解释的关键词证据，提升视觉语言模型分析故障能力。
KITE: Keyframe-Indexed Tokenized Evidence for VLM-Based Robot Failure Analysis
- 用关键帧+鸟瞰图编码运动和布局信息，生成结构化提示
- 在仿真任务中故障检测与定位准确率显著提升，优于原版模型
- 无需训练即可部署，适合实际机器人系统故障分析
我们提出KITE，一种无需训练、以关键帧锚定、布局对齐的前端框架，将长时机器人执行视频转化为紧凑且可解释的视觉-语言模型（VLM）输入。KITE将每条轨迹压缩为少量运动显著的关键帧，包含开放词汇检测结果，并为每个关键帧配以编码相对物体布局、坐标轴、时间戳和置信度的鸟瞰图（BEV）表示。这些视觉线索与机器人属性及场景上下文标记串联成统一提示，使同一前端支持故障检测、识别、定位、解释与纠正，仅需现成VLM。在RoboFAC基准上，KITE结合Qwen2.5-VL在无训练设置下表现优于原始模型，尤其在仿真故障检测、识别与定位上提升明显，同时媲美经微调的基线。小规模QLoRA微调进一步提升解释与纠正质量。我们在真实双臂机器人上也展示了定性效果，验证了KITE作为结构化、可解释前端在机器人故障分析中的实用性。代码与模型已开源：https://m80hz.github.io/kite/
原文摘要 · Abstract (English)
We present KITE, a training-free, keyframe-anchored, layout-grounded front-end that converts long robot-execution videos into compact, interpretable tokenized evidence for vision-language models (VLMs). KITE distills each trajectory into a small set of motion-salient keyframes with open-vocabulary detections and pairs each keyframe with a schematic bird's-eye-view (BEV) representation that encodes relative object layout, axes, timestamps, and detection confidence. These visual cues are serialized with robot-profile and scene-context tokens into a unified prompt, allowing the same front-end to support failure detection, identification, localization, explanation, and correction with an off-the-shelf VLM. On the RoboFAC benchmark, KITE with Qwen2.5-VL substantially improves over vanilla Qwen2.5-VL in the training-free setting, with especially large gains on simulation failure detection, identification, and localization, while remaining competitive with a RoboFAC-tuned baseline. A small QLoRA fine-tune further improves explanation and correction quality. We also report qualitative results on real dual-arm robots, demonstrating the practical applicability of KITE as a structured and interpretable front-end for robot failure analysis. Code and models are released on our project page: https://m80hz.github.io/kite/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。