纯视觉导航网页，性能超越大厂系统。
WebSight: A Vision-First Architecture for Robust Web Agents
- 用视觉模型直接理解网页界面，无需解析代码。
- 在关键任务上达到68.0%成功率，优于OpenAI和HCompany方案。
- 高精度响应（97.14%正确率），适合追求可靠性的自动化场景。
我们提出WebSight，一种基于视觉的自主网页代理，通过纯视觉感知与网页环境交互，摆脱对HTML或DOM输入的依赖。核心是训练有素的视觉语言模型WebSight-7B，基于Wave-UI-25K数据集的网页子集，使用LoRA微调。WebSight采用模块化多代理架构，包含规划、推理、视觉动作与验证代理，并通过情景记忆机制协同工作。WebSight-7B在Showdown Clicks基准上实现58.84%的顶级准确率，优于多个更大规模通用模型且延迟更低。完整WebSight代理在WebVoyager基准上达到68.0%成功度，超过OpenAI（61.0%）和HCompany（Runner H, 67.0%）。完成任务中，其正确回答率达97.14%，体现高精度。WebSight与WebSight-7B共同树立了可解释、鲁棒且高效的视觉网页导航新标准。
原文摘要 · Abstract (English)
We introduce WebSight, a vision-based autonomous web agent, designed to interact with web environments purely through visual perception, eliminating dependence on HTML or DOM-based inputs. Central to our approach we introduce our new model, WebSight-7B, a fine-tuned vision-language model optimized for UI element interaction, trained using LoRA on a web-focused subset of the Wave-UI-25K dataset. WebSight integrates this model into a modular multi-agent architecture, comprising planning, reasoning, vision-action, and verification agents, coordinated through an episodic memory mechanism. WebSight-7B achieves a top-1 accuracy of 58.84% on the Showdown Clicks benchmark, outperforming several larger generalist models while maintaining lower latency. The full WebSight agent achieves a 68.0% success rate on the WebVoyager benchmark, surpassing systems from labs such as OpenAI (61.0%) and HCompany (Runner H, 67.0%). Among tasks completed, WebSight answers correctly 97.14% of the time, indicating high precision. Together, WebSight and WebSight-7B establish a new standard for interpretable, robust, and efficient visual web navigation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。