arXiv:2608.09944cs.HCcs.AI2026-08

评测助盲UI代理的导航与解释能力,发现顶尖模型表现仍不佳。

Navigation Alone Is Not Enough: Evaluating Explanatory Assistive UI Agents

  • 设计新基准NeXUI,要求代理边导航边用自然语言解释每步操作。
  • 顶尖模型如Gemini-3.5-Flash仅44%任务成功率,解释质量差。
  • 适合研究无障碍交互、可解释AI代理的学者与开发者。

现代网页界面因动态更新和视觉布局隐藏结构,对屏幕阅读器越来越不友好。尽管现有UI代理能执行操作,但真正有用的辅助代理应作为协作伙伴,持续告知用户并保持其控制权。当前多数评估只关注任务完成度,忽视解释性和用户监督。我们提出NeXUI基准,要求代理在真实用户目标下导航,并用清晰语言解释每一步,同时利用可视化上下文和结构信息推理。评估涵盖安全性、效率、任务成功率,还验证解释是否基于界面状态。实验显示,即使最先进的基础模型(如Gemini-3.5-Flash)在该基准上也仅达44%成功率,且解释得分低,表明其仍具挑战性,可为未来研究提供可靠基础。通过聚焦导航、解释与用户控制,NeXUI为支持视障用户的智能代理研究提供了更清晰路径。

原文摘要 · Abstract (English)

Modern web interfaces are increasingly difficult to use with screen readers, particularly when pages update dynamically or hide important structure behind visual layout. Recent UI agents can act on such interfaces; however, for assistive agents to be truly useful, they must behave as collaborators that keep users informed and in control, rather than as tools that simply take actions on users' behalf. Most existing benchmarks judge systems primarily by task completion, without assessing how well they explain their actions or support user oversight. We introduce NeXUI, a benchmark for assistive agents that must navigate interfaces while explaining each step in clear language for nonvisual use. NeXUI pairs realistic user goals with instrumented interface states, enabling agents to reason from both visual context and structural information. Its evaluation measures safety, efficiency, and task success, while also checking whether explanations are grounded in the interface state. In our experiments, we find that NeXUI remains challenging even for state-of-the-art foundation models, with % Gemini-3.5-Flash achieving only a 44\% success rate and poor explanation scores, making it a useful foundation for future research and development. By focusing on navigation, explanation, and user control, NeXUI provides a clearer way to study agents that can support blind and visually impaired users in modern computing environments.

UI代理无障碍解释性AI

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。