arXiv:2601.21961cs.AIcs.HC2026-01ACL被引 4

探究网页视觉属性如何影响网络代理决策,发现颜色对比和布局位置最关键。

How do Visual Attributes Influence Web Agents? A Comprehensive Evaluation of User Interface Design Factors

  • 构建可控评估流程VAF,分离视觉属性与语义信息
  • 背景色对比、物品大小与位置显著影响点击率,字体等影响较小
  • 适合研究网页设计对AI代理行为影响的开发者与研究人员

网络代理在多种网页任务中表现出色,但现有研究多关注对抗攻击下的鲁棒性,对良性环境下代理偏好关注不足。尽管早期研究探讨过文本属性的影响,但对视觉属性如何塑造代理决策尚缺乏系统理解。为此,本文提出VAF评估流程,通过三阶段量化网页视觉属性因素对代理决策的影响:(i) 变体生成,保持语义一致仅改变视觉属性;(ii) 浏览交互,模拟人类用户滚动与点击行为;(iii) 通过点击动作与推理结果验证,使用目标点击率(Target Click Rate)与目标提及率(Target Mention Rate)联合评估。在8类变体(共48个)、5个真实网站(含购物、旅游、新闻)及4个代表性网络代理上进行大量实验,结果表明:背景色对比、物品大小、位置和卡片清晰度对代理行为有显著影响,而字体样式、文字颜色和图片清晰度影响较小。

原文摘要 · Abstract (English)

Web agents have demonstrated strong performance on a wide range of web-based tasks. However, existing research on the effect of environmental variation has mostly focused on robustness to adversarial attacks, with less attention to agents' preferences in benign scenarios. Although early studies have examined how textual attributes influence agent behavior, a systematic understanding of how visual attributes shape agent decision-making remains limited. To address this, we introduce VAF, a controlled evaluation pipeline for quantifying how webpage Visual Attribute Factors influence web-agent decision-making. Specifically, VAF consists of three stages: (i) variant generation, which ensures the variants share identical semantics as the original item while only differ in visual attributes; (ii) browsing interaction, where agents navigate the page via scrolling and clicking the interested item, mirroring how human users browse online; (iii) validating through both click action and reasoning from agents, which we use the Target Click Rate and Target Mention Rate to jointly evaluate the effect of visual attributes. By quantitatively measuring the decision-making difference between the original and variant, we identify which visual attributes influence agents' behavior most. Extensive experiments, across 8 variant families (48 variants total), 5 real-world websites (including shopping, travel, and news browsing), and 4 representative web agents, show that background color contrast, item size, position, and card clarity have a strong influence on agents' actions, whereas font styling, text color, and item image clarity exhibit minor effects.

网页代理视觉属性评估框架人机交互

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。