arXiv:2602.02548cs.LGcs.AI2026-02被引 1

用可学习的工具标记实现高效通用的GUI操作,减少数据依赖。

ToolTok: Tool Tokenization for Efficient and Generalizable GUI Agents

  • 将操作拆解为逐步使用工具序列,避免坐标依赖。
  • 仅用不到1%的数据量,性能超越40亿参数模型。
  • 适合需要少样本训练和跨场景泛化的GUI自动化研究者。

现有基于坐标的单步视觉定位的GUI代理模型在不同分辨率和长宽比下泛化能力差。替代方法虽采用无坐标策略,但面临严重数据稀缺问题。为此,我们提出ToolTok,一种面向GUI代理的多步路径规划新范式,将操作建模为逐步使用工具的序列。具体地,设计符合人类交互习惯的工具,并用可学习的标记嵌入表示每个工具。为在有限监督下高效学习嵌入,引入语义锚定机制,以语义相关概念作为自然归纳偏置来定位工具。为进一步使预训练大语言模型逐步掌握工具语义,构建了由易到难的三阶段课程:标记定义问答、纯文本引导工具选择、简化视觉路径规划。在多个基准上的实验表明,ToolTok在同等规模(40亿参数)模型中表现最优,且优于更大规模(2350亿参数)模型,仅需其他后训练方法1%以下的训练数据。此外,其在未见场景中展现出强泛化能力。训练与推理代码已开源:https://github.com/ZephinueCode/ToolTok。

原文摘要 · Abstract (English)

Existing GUI agent models relying on coordinate-based one-step visual grounding struggle with generalizing to varying input resolutions and aspect ratios. Alternatives introduce coordinate-free strategies yet suffer from learning under severe data scarcity. To address the limitations, we propose ToolTok, a novel paradigm of multi-step pathfinding for GUI agents, where operations are modeled as a sequence of progressive tool usage. Specifically, we devise tools aligned with human interaction habits and represent each tool using learnable token embeddings. To enable efficient embedding learning under limited supervision, ToolTok introduces a semantic anchoring mechanism that grounds each tool with semantically related concepts as natural inductive bias. To further enable a pre-trained large language model to progressively acquire tool semantics, we construct an easy-to-hard curriculum consisting of three tasks: token definition question-answering, pure text-guided tool selection, and simplified visual pathfinding. Extensive experiments on multiple benchmarks show that ToolTok achieves superior performance among models of comparable scale (4B) and remains competitive with a substantially larger model (235B). Notably, these results are obtained using less than 1% of the training data required by other post-training approaches. In addition, ToolTok demonstrates strong generalization across unseen scenarios. Our training & inference code is open-source at https://github.com/ZephinueCode/ToolTok.

GUI代理工具标记少样本学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。