用视觉界面让通用大模型直接控制机器人，零样本完成复杂操作。
VIA: Visual Interface Agent for Robot Control
- 将机器人控制转为智能体任务，通过浏览器界面用截图和指令交互。
- Fable 5模型在LIBERO-Goal任务上达96.7%成功率，彩虹组装任务100%成功。
- 无需微调，仅靠通用大模型能力即可实现规划、纠错与重规划。
机器人操作涉及视觉理解、物理推理、规划与闭环控制。当前方法通常通过在机器人数据上微调通用基础模型（FMs），将其转化为视觉-语言-动作（VLA）模型以输出低层动作，但受限于数据与算力，这类VLA规模远小于前沿基础模型，制约其泛化能力。受基础模型操作软件界面能力的启发，我们提出VIA（Visual Interface Agent），将机器人控制重构为智能体任务：一个现成的FM驱动机械臂，通过浏览器中的3D界面进行操作，包括截图、下达自然指令、观察结果并调整策略。该代理不接受任何机器人特定微调，也无特权状态信息，仅依赖视觉输入与一组通用工具。VIA继承了智能体的通用推理、闭环纠错及动态规划能力。在Claude Code与Codex支持下，它实现了对多种桌面操作任务的零样本求解。使用最强模型Fable 5时，在三个LIBERO-Goal任务上达到96.7%成功率，在长时程彩虹装配任务上实现100%成功。性能随底层模型规模与能力提升而增强。结果表明，前沿智能体已具备可直接迁移至机器人控制的核心技能——你的编程或计算机使用智能体，本质上已是机器人控制智能体。
原文摘要 · Abstract (English)
Robot manipulation is a complex task that requires visual understanding, physical reasoning, planning, and closed-loop control. General-purpose foundation models (FMs) have grown remarkably capable of some of these, especially vision and reasoning. To leverage this for generalist robot policies, current methods typically involve converting existing FMs into vision-language-action (VLA) models by fine-tuning on robot data to output low-level actions. However, VLAs are often orders of magnitude smaller than frontier FMs given the limited data and compute available for fine-tuning, which in turn limits their general capability. Inspired by the growing ability of FMs to operate software through visual interfaces, we ask whether that same competence suffices to control a robot. We present VIA (Visual Interface Agent for robot control), a framework that recasts robot control as an agentic task: an off-the-shelf FM-powered agent drives a manipulator through a browser-based 3D interface by taking screenshots, issuing intuitive commands, observing the outcome, and adjusting. The agent receives no robot-specific fine-tuning and no access to privileged state information: it perceives visual input and acts through a small set of general tools. VIA inherits the agent's general reasoning, closed-loop error recovery, and ability to plan and re-plan from what it observes. It solves a diverse suite of tabletop manipulation tasks zero-shot with both Claude Code and Codex. With the strongest model (Fable 5) it achieves 96.7% success on three LIBERO-Goal tasks and 100% on a long-horizon rainbow assembly task. Performance improves with the scale and strength of the underlying model. These results suggest that frontier agents already possess skills that transfer directly to robot control given the right interface: your coding or computer-use agent is, in a sense, secretly a robot-control agent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。