用离线强化学习训练视觉语言模型的行动价值函数，实现无需真实交互的设备控制智能体。
Digi-Q: Learning Q-Value Functions for Training Device-Control Agents
- 基于冻结的视觉语言模型特征，用离线时序差分学习训练行动价值函数。
- 在安卓设备控制任务中比最优基线提升21.2%，接近需交互的先进强化学习方法。
- 适合需要低成本、高安全性的开放场景智能体训练，如手机操作自动化。
现有构建基础模型智能体的方法多依赖提示或人类示范，但在动态环境（如移动设备控制）中不足。在线强化学习虽可应对，但真实环境中的交互成本过高。为此，我们提出Digi-Q，通过离线时序差分学习，在冻结的视觉语言模型中间层特征基础上训练行动价值函数（Q函数），避免全模型微调带来的计算开销。为使模型特征适配价值函数表示，先进行初步微调以增强对动作相关信息的覆盖。训练完成后，采用“最佳候选动作”策略提取器，从多个候选动作中选出由价值函数排序的最佳动作，实现无环境交互的策略优化。在Android-in-the-Wild用户规模设备控制任务中，Digi-Q相比先前最优方法提升21.2%，部分情况下已达到需交互的先进强化学习方法水平。项目已开源于https://github.com/DigiRL-agent/digiq。
原文摘要 · Abstract (English)
While a number of existing approaches for building foundation model agents rely on prompting or fine-tuning with human demonstrations, it is not sufficient in dynamic environments (e.g., mobile device control). On-policy reinforcement learning (RL) should address these limitations, but collecting actual rollouts in an environment is often undesirable in truly open-ended agentic problems such as mobile device control or interacting with humans, where each unit of interaction is associated with a cost. In such scenarios, a method for policy learning that can utilize off-policy experience by learning a trained action-value function is much more effective. In this paper, we develop an approach, called Digi-Q, to train VLM-based action-value Q-functions which are then used to extract the agent policy. We study our approach in the mobile device control setting. Digi-Q trains the Q-function using offline temporal-difference (TD) learning, on top of frozen, intermediate-layer features of a VLM. Compared to fine-tuning the whole VLM, this approach saves us compute and enhances scalability. To make the VLM features amenable for representing the Q-function, we need to employ an initial phase of fine-tuning to amplify coverage over actionable information needed for value function. Once trained, we use this Q-function via a Best-of-N policy extraction operator that imitates the best action out of multiple candidate actions from the current policy as ranked by the value function, enabling policy improvement without environment interaction. Digi-Q outperforms several prior methods on user-scale device control tasks in Android-in-the-Wild, attaining 21.2% improvement over prior best-performing method. In some cases, our Digi-Q approach already matches state-of-the-art RL methods that require interaction. The project is open-sourced at https://github.com/DigiRL-agent/digiq
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。