用语音控制网页导航，让残障用户更自由地访问网络
WebNav: An Intelligent Agent for Voice-Controlled Web Navigation
- 双大模型分工：一个规划策略，一个生成具体操作
- 结合截图与页面标签识别按钮等交互元素
- 适合无障碍浏览、智能自动化场景的开发者参考
当前网页界面在无障碍使用方面存在严重不足。传统方法如脚本语言和屏幕阅读器难以应对动态内容，也无法理解用户的高层次目标。为此，我们提出WebNav，一种多模态网页导航智能体。该系统采用双大语言模型架构，将自然语言指令转化为图形界面的精确可执行动作。通过截图获取视觉上下文，并结合动态DOM标签浏览器扩展，稳健识别可交互元素。高层‘控制器’模型规划下一步行动，低层‘助手’模型生成具体执行参数。这种分工使任务分解与动作生成更加复杂精细。本文完整展示了WebNav的架构与实现，为构建更智能的网页自动化代理提供了可行路径。
原文摘要 · Abstract (English)
The current state of modern web interfaces, especially in regards to accessibility focused usage is extremely lacking. Traditional methods for web interaction, such as scripting languages and screen readers, often lack the flexibility to handle dynamic content or the intelligence to interpret high-level user goals. To address these limitations, we introduce WebNav, a novel agent for multi-modal web navigation. WebNav leverages a dual Large Language Model (LLM) architecture to translate natural language commands into precise, executable actions on a graphical user interface. The system combines vision-based context from screenshots with a dynamic DOM-labeling browser extension to robustly identify interactive elements. A high-level 'Controller' LLM strategizes the next step toward a user's goal, while a second 'Assistant' LLM generates the exact parameters for execution. This separation of concerns allows for sophisticated task decomposition and action formulation. Our work presents the complete architecture and implementation of WebNav, demonstrating a promising approach to creating more intelligent web automation agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。