用轻量级Q函数在推理时重排动作，提升视觉语言模型代理的网页操作成功率。
Best-of-Q: Improving VLM agents with Q-function Action Ranking at Inference
- 冻结VLM模型，用Q函数在推理时重排序候选动作。
- 使Qwen2.5-VL-7B成功率从38.8%提升至55.7%，GPT-4.1从82.4%升至88.8%。
- 无需重新训练模型，适合快速部署于动态网络环境的智能代理。
视觉语言模型（VLM）已成为自主操作数字环境（如网络和操作系统）的智能体核心。然而，这些模型难以适应快速变化的网络环境，传统改进方法依赖大规模微调，需大量训练和数据收集。本文提出一种新型推理阶段增强策略：将VLM作为高容量动作生成器，保持其冻结状态，生成候选动作集；再由一个离线训练的轻量级Q函数对候选动作进行重排序，选择价值最高的动作执行。该方法直接在推理阶段应用Q函数实现即时性能提升，而非用于数据重标注以重新训练策略。在学术基准WebVoyager上验证，Qwen2.5-VL-7B代理成功率从38.8%提升至55.7%，专有GPT-4.1代理从82.4%提升至88.8%。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have become powerful backbones for agents to autonomously operate in digital environments like the web and operating systems. However, these models suffer from inadaptability to fast-changing environments like the web, which can be alleviated by fine-tuning requiring expansive model training and data collection. In this work, we introduce a novel paradigm for enhancing agentic VLM policies at inference without policy retraining. Fundamentally, our approach decouples the VLM's role as a high-capacity action proposer from the final action selection mechanism. We keep the VLM policy frozen and use it to generate a set of candidate actions for a given state. Then, a lightweight, offline-trained Q-function reranks these candidates, and the agent executes the action with the highest estimated value. The main contribution is to apply the Q-function directly during inference for immediate policy improvement, and not offline to relabel data for policy retraining. We demonstrate on the academic WebVoyager benchmark that our method significantly boosts agent success rates, improving a Qwen2.5-VL-7B agent from 38.8% to 55.7% and a proprietary GPT-4.1 agent from 82.4% to 88.8%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。