用图像修复技术让机器人仅靠视觉实现无模型控制，即使被遮挡也能持续追踪关键点。
Utilizing Inpainting for Keypoint Detection for Vision-Based Control of Robotic Manipulators
- 通过图像修复移除标记物并自动标注关键点，无需标定或精确模型。
- 在完全可见和部分遮挡下均实现稳定关键点检测与控制。
- 适合缺乏先验模型的机器人系统，尤其适用于柔性可折叠机械臂。
我们提出一种新型视觉伺服框架，仅使用自然视觉特征在配置空间中控制机械臂。训练数据驱动的关键点检测器时，在机器人本体上粘贴ArUco标记，以其中心作为关键点标签，并利用图像修复技术移除标记并重建被遮挡区域，从而生成无需人工标注、无标记的机器人图像，且不依赖精确相机标定或机器人模型。运行时，第二个修复模型重建部分遮挡区域，实现连续关键点检测。无迹卡尔曼滤波器（UKF）进一步提升关键点估计的时间一致性与鲁棒性。我们在全视野和部分遮挡条件下均实现了成功的无模型视觉控制。为验证泛化能力，还将感知流程扩展至两模块和三模块软折纸臂，并定性评估了关键点检测与时间跟踪效果。
原文摘要 · Abstract (English)
We present a novel visual servoing framework for controlling a robotic manipulator in configuration space using only natural visual features. To train our data-driven keypoint detector, we attach ArUco markers along the robot body, use their centers as keypoint labels, and apply image inpainting to remove the markers and reconstruct the occluded regions. This produces automatically labeled, markerless robot images without requiring accurate camera calibration or robot models. At runtime, a second inpainting model reconstructs robot regions that are partially occluded, enabling continuous keypoint detection. An Unscented Kalman Filter (UKF) further improves temporal consistency and robustness of the keypoint estimates. We demonstrate successful model-free, vision-based control using natural robot features under both full visibility and partial occlusion. To show broader applicability, we also extend the perception pipeline to two-module and three-module soft origami arms and qualitatively evaluate keypoint detection and temporal tracking on these platforms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。