用文字版图像让纯文本大模型控制机器人，省算力还易懂。
ASCII Art Turns LLMs into VLA Controllers
- 把图像转成ASCII字符文本，让纯文本大模型处理视觉输入。
- 在2D操作任务中能识别关键物体并规划可行动作序列。
- 适合想用轻量级文本模型做视觉-语言-动作控制的研究者。
视觉-语言-动作（VLA）控制器通常通过在视觉-语言模型（VLM）上添加动作监督构建，依赖大规模数据和计算资源的多模态骨干网络。本文展示，当视觉观测通过ASCII表示转化为文本输入时，纯文本大语言模型（LLM）可被适配为具备VLA风格的控制器。该ASCII视觉接口使现有LLM的训练与部署流程能够高效地基于视觉状态进行条件化，遵循自然语言指令，并生成受约束、可执行的动作。我们在多个模型家族和规模下微调并比较了多种LLM与VLM，使用基于规划的教师提供的专家示范，以及DAgger实现迭代优化。在2D操作基准测试中，无论是在仿真环境还是物理机械臂上，所得到的控制器均能识别任务相关实体并规划可行的动作序列。结果表明，ASCII渲染可作为从图像到文本的轻量、可解释的模态桥梁，补充传统VLA流水线，并为基于纯文本骨干的VLA研究开辟新方向。
原文摘要 · Abstract (English)
Vision--Language--Action (VLA) controllers are often built by extending vision--language models (VLMs) with action supervision, relying on multimodal backbones with large data and compute requirements. We demonstrate that a text-only large language model (LLM) can be adapted into a VLA-style controller when visual observations are rendered into a text input using an ASCII representation. This ASCII-as-vision interface enables existing training and deployment stacks for LLMs to efficiently condition on visual state, follow natural-language instructions, and produce constrained, executable actions. We fine-tune and compare multiple LLMs and VLMs across model families and scales, using both expert demonstrations from a planning-based teacher, as well as DAgger for iterative improvement. In a 2D manipulation benchmark, in both simulation and on a physical manipulator, the resulting controllers can identify task-relevant entities and plan feasible action sequences. Our results suggest that ASCII rendering can serve as a lightweight, interpretable modality bridge from images to text, complementing conventional VLA pipelines, and opening directions for VLA research with text-only backbones.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。