arXiv:2606.13192cs.AI2026-06中稿 · CVPR

构建首个界面推理评测基准,提升大模型对用户体验的分析能力

Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach

论文配图:Reasoning for Mobile User Experience with Multimodal LLMs: Task, Benchmark, and Approach
图 1 · 摘自论文原文
  • 设计8类真实界面任务,评估模型对布局、层级、一致性等细节的推理能力
  • 提出UI-UX模型在新基准上达0.7963准确率,超越Claude-4.5-Sonnet的0.6550
  • 通过动态奖励路由与不对称奖励机制,实现高效精准的推理决策

以可用性、感知一致性和功能清晰性为核心的用户体验(UX)是现实用户界面(UI)的基础。多模态大语言模型(MLLMs)在界面领域的应用迅速发展,涵盖视觉元素定位、图形界面代理及设计转代码生成等。然而,基于界面截图评估用户体验的研究仍不成熟。为此,我们提出UXBench——一个包含2,000个VQA样本的新多模态基准,用于评估MLLMs在界面推理方面的能力。UXBench包含8类基于真实界面截图的任务,需细粒度诊断布局关系、视觉层次与内容一致性问题。对主流MLLMs的广泛评估显示,其在界面推理能力上仍存在根本性局限。为填补这一差距,我们提出UI-UX,基于Qwen3-VL-4B-Thinking基础模型,并通过强化学习改进:引入奖励路由机制,在推理中动态平衡感知理解与逻辑推理;采用非对称转移奖励,抑制冗余或不足的推理步骤。实验表明,UI-UX在UXBench上达到0.7963的准确率,显著优于Claude-4.5-Sonnet的0.6550,同时具备强泛化能力且推理延迟低。

原文摘要 · Abstract (English)

User experience (UX) centered on usability, perceived consistency, and functional clarity is fundamental to real-world user interfaces (UI). The application of multimodal large language models (MLLMs) in the field of user interfaces is evolving rapidly, such as visual element grounding, graphical user interface (GUI) agents, and design-to-code generation. However, research efforts on evaluating UX based on UI screenshots are still immature. To address this, we propose UXBench, a novel multimodal benchmark consisting of 2,000 VQA data samples designed to assess MLLMs' ability to perform UI-based reasoning. UXBench includes 8 tasks based on real-world UI screenshots that require fine-grained diagnosis of UX issues across layout relationships, visual hierarchy, and content consistency. Our extensive evaluation of mainstream MLLMs shows that they remain fundamentally limited in their capacity for UI-based reasoning. The results underscore the need for further advancements in this area. To bridge this gap, we propose UI-UX, an MLLM based on Qwen3-VL-4B-Thinking foundation model and enhanced via reinforcement learning with two key innovations: a reward routing mechanism that dynamically balances perceptual understanding and logical reasoning during inference, and an asymmetric transition reward that suppresses redundant or insufficient reasoning steps. Experiments demonstrate that UI-UX achieves state-of-the-art (SOTA) performance on UXBench, attaining an accuracy of 0.7963 -- surpassing Claude-4.5-Sonnet's 0.6550 -- while exhibiting strong generalization across diverse UI tasks and maintaining low inference latency.

多模态大模型用户体验界面推理评测基准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。