让大模型在有限操作次数内可靠抓取网页图文视频数据
BaRA: Budget-constrained and Reliable Web Data Collection Agent
- 用广度优先搜索+存活验证，精准发现有效链接
- 通过溯源与可访问性检查，确保提取的图文视频可用
- 自反思机制能修复失败任务，适合真实网页采集
基于大语言模型的网络代理可自动化网页导航与数据采集。然而，实时网页数据采集不仅需完成任务，还需在固定交互预算内发现站点内页面，并以可访问形式获取文本、图像和视频内容。本文将此场景定义为受限预算的站点级多模态网页数据采集，提出预算约束且可靠的代理（BaRA）。BaRA采用基于广度优先搜索的链接发现方法，并结合存活验证过滤幻觉与失效链接；通过规则化溯源与可访问性检查验证提取的多模态内容；引入基于历史的自反思模块，恢复执行失败与不完整输出。在受控的合成网站与真实网站上，BaRA在有效链接发现率和下载有效多模态内容方面均优于现有代理。代码已开源：https://github.com/MLAI-Yonsei/BaRA-Agent。
原文摘要 · Abstract (English)
Large language model (LLM)-based web agents automate web navigation and data collection. However, live web data collection demands capabilities beyond task completion: agents must discover site-internal pages and retrieve text, image, and video artifacts in an accessible form within a fixed interaction budget. We formulate this setting as budget-constrained, site-level multimodal web data collection and propose Budget-constrained and Reliable Agent (BaRA). BaRA performs breadth-first search (BFS)-based link discovery with liveness verification to filter hallucinated and dead links, then validates extracted multimodal artifacts using rule-based provenance and accessibility checks. A history-based self-reflection module recovers from execution failures and incomplete outputs. On controlled synthetic and real-world websites, BaRA consistently improves valid-link discovery and download-valid multimodal extraction over existing agents. Our code is available at https://github.com/MLAI-Yonsei/BaRA-Agent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。