用预训练视觉语言模型实现机器人环境状态统一识别，无需训练或编程。
Robotic Environmental State Recognition with Pre-Trained Vision-Language Models and Black-Box Optimization
- 通过视觉问答和图文检索任务，用自然语言描述实现环境状态识别。
- 识别准确率可通过黑盒优化选择最佳文本提升，支持透明门和流水等复杂场景。
- 只需更换文本集与权重，即可适应多种状态，适合快速部署的机器人系统。
为使机器人在多样化环境中自主导航与操作，环境状态识别至关重要。传统方法针对每种状态需定制不同算法。本文提出一种基于预训练大规模视觉-语言模型的统一环境状态识别方法，利用视觉问答（Visual Question Answering）和图像到文本检索（Image-to-Text Retrieval）任务，仅通过自然语言描述即可识别房间门是否开启、透明门状态及水槽是否流水，无需神经网络训练或人工编程。通过黑盒优化从预设文本集中选取最优文本组合，可进一步提升识别准确率。每种状态识别仅需调整文本集及其权重，无需构建多个独立模型与程序，显著简化代码管理与资源占用。实验验证了该方法的有效性，并在移动机器人Fetch上实现了实际应用。
原文摘要 · Abstract (English)
In order for robots to autonomously navigate and operate in diverse environments, it is essential for them to recognize the state of their environment. On the other hand, the environmental state recognition has traditionally involved distinct methods tailored to each state to be recognized. In this study, we perform a unified environmental state recognition for robots through the spoken language with pre-trained large-scale vision-language models. We apply Visual Question Answering and Image-to-Text Retrieval, which are tasks of Vision-Language Models. We show that with our method, it is possible to recognize not only whether a room door is open/closed, but also whether a transparent door is open/closed and whether water is running in a sink, without training neural networks or manual programming. In addition, the recognition accuracy can be improved by selecting appropriate texts from the set of prepared texts based on black-box optimization. For each state recognition, only the text set and its weighting need to be changed, eliminating the need to prepare multiple different models and programs, and facilitating the management of source code and computer resource. We experimentally demonstrate the effectiveness of our method and apply it to the recognition behavior on a mobile robot, Fetch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。