arXiv:2606.28344cs.IRcs.AI2026-06被引 2

用网页截图代替文字,让大模型检索更准更快

PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation

论文配图:PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation
图 1 · 摘自论文原文
  • 直接在像素层面处理网页截图,跳过文本解析环节
  • 在3000万张截图上实现高效检索,多任务性能提升最高18.1%
  • 适合追求高精度、低延迟的网页问答与智能体应用

将大语言模型与检索增强结合已成为主流范式,但网络内容并非天然为文本:现有系统依赖复杂的解析流程,线性化HTML并丢弃布局、视觉结构和格式。本文提出PixelRAG,一种全新的检索增强方法,以网页原始视觉形式表示内容,在像素空间中完成检索与阅读,实现端到端架构,消除文本抽象。PixelRAG是首个在完整维基百科语料库上以该形式运行的系统,支持3000万张截图构成的数据存储,并构建了高效的视觉检索索引。基于现有视觉嵌入模型(Qwen3-VL-Embedding),通过精心设计的对比学习数据对截图进行微调。检索到的截图直接以像素输入至视觉语言模型,无需中间文本转换。PixelRAG在多项任务中持续优于无检索及文本基基线,尤其在广泛研究的文本密集型任务(如NQ、SimpleQA)表现突出。同时在多模态开放域问答(如MMSearch)、噪声新闻语料(如LiveVQA)和智能体基准(如MoNaCo)上取得显著提升,准确率最高比文本基线上升18.1%。此外,像素表示支持图像压缩新策略,可在较低分辨率下实现高达3倍的令牌成本降低,同时保持精度。结果表明,网页检索无需依赖文本表示,可直接以网页原生视觉形式运行,同时提升性能与效率。

原文摘要 · Abstract (English)

Augmenting large language models (LLMs) with retrieved web text has become a dominant paradigm, yet the web is not natively textual: existing systems depend on complex parsing pipelines that linearize HTML and discard layout, visual structure, and formatting. We introduce PixelRAG, a new retrieval-augmented method that represents websites in their native visual form and performs retrieval and reading entirely in pixel space, enabling an end-to-end architecture that eliminates text abstraction. PixelRAG is, to our knowledge, the first pipeline to operate over a full Wikipedia corpus in this form, scaling to a datastore of 30 million screenshot images with an efficient visual retrieval index. Built on an existing visual embedding model (i.e., Qwen3-VL-Embedding), PixelRAG further fine-tunes this model on screenshot data with carefully curated contrastive training data. Retrieved screenshots are then fed directly as pixel inputs to a VLM, without intermediate text conversion. PixelRAG consistently outperforms both no-retrieval and text-based RAG baselines, most surprisingly on widely studied text-centric tasks such as NQ and SimpleQA. It also achieves strong gains on multimodal open-domain QA (e.g., MMSearch), benchmarks over noisy news corpora (e.g., LiveVQA), and agentic benchmarks (e.g., MoNaCo), improving accuracy by up to 18.1% over text-based baselines. Finally, pixel representations enable a new efficiency lever for RAG through image compression, achieving up to 3x token cost reduction at lower resolutions while maintaining accuracy. Our results challenge the necessity of text representations in web retrieval, suggesting that web RAG can operate directly in the web's native visual form while improving both performance and efficiency.

视觉检索网页理解RAG优化多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。