arXiv:2509.21782cs.AI2025-09被引 9

评测多模态大模型在网页理解中的推理、鲁棒性与安全性表现。

Benchmarking MLLM-based Web Understanding: Reasoning, Robustness and Safety

  • 构建涵盖8个任务的综合评测基准WebRRSBench,覆盖多步推理与安全交互。
  • 11个MLLM模型在真实网页上测试,普遍存在跨元素推理弱、抗干扰能力差问题。
  • 适合研究网页智能体、自动化系统安全性的开发者和研究人员参考。

多模态大语言模型(MLLMs)正日益成为面向网络系统的核心推理引擎,支撑图形界面代理与前端自动化,需可靠解析页面结构、选择可操作控件并执行多步骤交互。然而现有评测主要聚焦视觉感知或UI代码生成,对端到端网页应用所需的推理、鲁棒性与安全性评估不足。为此,我们提出一个综合性网页理解评测基准WebRRSBench,联合评估推理、鲁棒性与安全性,覆盖8项任务,如位置关系推理、颜色鲁棒性、安全关键检测等。该基准基于729个真实网站,包含3799组问答对,用于探测对页面结构、文本、控件及安全关键交互的多步推理能力。为确保评估可靠性,采用标准化提示、协议化且确定性的评测流程,并结合自动检查与针对性人工验证进行多阶段质量控制。我们在WebRRSBench上评估了11个MLLM模型,结果揭示显著差距:模型在复杂布局上的组合式与跨元素推理仍显不足,面对界面扰动(如布局重排、视觉风格变化)时鲁棒性有限,且对安全关键或不可逆操作识别保守。代码与附录见https://github.com/annoy-worker/WebRSSBench。

原文摘要 · Abstract (English)

Multimodal large language models (MLLMs) are increasingly deployed as the core reasoning engine for web-facing systems, powering GUI agents and front-end automation that must interpret page structure, select actionable widgets, and execute multi-step interactions reliably. However, existing benchmarks largely emphasize visual perception or UI code generation, showing insufficient evaluation on the reasoning, robustness and safety capability required for end-to-end web applications. To bridge the gap, we introduce a comprehensive web understanding benchmark, named WebRRSBench, that jointly evaluates Reasoning, Robustness, and Safety across eight tasks, such as position relationship reasoning, color robustness, and safety critical detection, etc. The benchmark is constructed from 729 websites and contains 3799 QA pairs that probe multi-step inference over page structure, text, widgets, and safety-critical interactions. To ensure reliable measurement, we adopt standardized prompts, a protocolized and deterministic evaluation pipeline, and multi-stage quality control combining automatic checks with targeted human verification. We evaluate 11 MLLMs on WebRRSBench. The results reveal significant gaps: models still struggle with compositional and cross-element reasoning over realistic layouts, show limited robustness when facing perturbations in user interfaces and content such as layout rearrangements or visual style shifts, and are rather conservative in recognizing and avoiding safety critical or irreversible actions. Our code and appendix are available at https: //github.com/annoy-worker/WebRSSBench.

多模态网页理解安全评测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。