用压缩网页结构的方法让大模型更高效地处理网页任务。
Beyond Pixels: Exploring DOM Downsampling for LLM-Based Web Agents
- 提出D2Snap算法，通过保留关键操作信息压缩网页DOM结构。
- 压缩后平均仅用16.5%上下文空间，成功率提升至73%。
- 纯文本提示已足够，图像输入对效果帮助不大，适合资源受限场景。
大型语言模型（LLMs）推动了自主网页浏览代理的发展：给定一个网页任务和序列化的用户界面（UI）状态，LLM需逐步生成可执行的操作。核心挑战在于如何将UI状态有效序列化供LLM使用。当前主流方法依赖带视觉线索的图形界面（GUI）快照，因其输入令牌量小而受青睐。文档对象模型（DOM）快照（以HTML形式序列化）是另一候选方案，可利用LLM对HTML的解析能力，但其过大的令牌开销限制了实际部署。本文提出D2Snap算法，通过保留可操作性和区分性特征来下采样DOM。我们在从Online-Mind2Web中采样的数据集上，使用GPT-4o作为快照型代理评估该方法。原始DOM快照中42%超过128,000令牌的模型上下文窗口限制，而所有经D2Snap处理的快照均能容纳，平均上下文利用率仅为16.5%。相比基于GUI快照的基线67%成功率，本方案达73%（+5.8个百分点；95%置信区间 -13.6至+26.0；McNemar检验，p=0.47），排除超出11个百分点缺陷的单侧95%置信度。此外，图像输入对快照效用贡献有限；仅靠文本引导即可达到62%（-5.8个百分点；McNemar检验，p=0.37）。
原文摘要 · Abstract (English)
The advent of large language models (LLMs) has sparked an evolution of autonomous web browsing agents: given a web browsing task and serialised user interface (UI) state, an LLM is expected to suggest input actions that incrementally solve the given task. The central challenge lies in serialising UI state for LLMs. Web agents have increasingly relied on grounded graphical UI (GUI) snapshots - screenshots augmented with visual cues - favoured for their modest input token footprint. Document object model (DOM) snapshots, serialised as HTML, represent a compelling alternative that leverages previously demonstrated HTML interpretation capabilities of LLMs. Their excessive token footprint, however, has precluded reliable deployment with web agents to date. We propose D2Snap, an algorithm to downsample the DOM, premised on preserving actionability and actionability-discriminating features. We evaluate D2Snap-downsampled DOM snapshots using a snapshot-variant web agent (GPT-4o) on a dataset sampled from Online-Mind2Web. Whilst 42% of raw DOM snapshots exceed the model context window (128 x 10^3 tokens), all D2Snap-downsampled DOM snapshots of our reference configuration fit, at a mean context utilisation of 16.5%. Against the 67% success rate of a grounded GUI snapshot baseline, our configuration attains 73% (+5.8%pt; 95% CI -13.6 to +26.0%pt; McNemar, p = 0.47), excluding a deficit (one-sided 95%) beyond 11%pt. Image input moreover appears to add little to snapshot utility; grounding text alone attains 62% (-5.8%pt; McNemar, p = 0.37).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。