用强化学习训练轻量级模型,自动识别生成网页的界面设计缺陷。
Learning to Detect UI Principle Violations via Reinforcement Learning

- 用合成注入法构建1万张带缺陷的网页数据,训练视觉语言模型做界面评判
- 微F1从36%提升至84%,13项设计原则超过80%准确率
- 适合自动化测试生成代码质量,尤其关注可访问性和用户体验
小型语言模型和编程代理生成的前端代码常仅通过功能校验,但可能违反界面质量准则,如无障碍障碍、误导性设计、视觉层级混乱及决策复杂度过高。现有审计方法在成本、覆盖范围与可扩展性间存在权衡:人工评审耗时昂贵,前沿视觉-语言模型部署成本高,而axe-core、Lighthouse等规则工具仅能检测机械性可访问问题。本文探索轻量级视觉-语言模型作为生成界面的评估者。整合了来自WCAG 2.2、欺骗性设计分类体系及感知认知理论的19项界面质量原则。通过在LLM生成的Tailwind页面中合成注入已知缺陷,构建约10,000个经验证的生成网页数据集。在40亿参数视觉-语言模型上持续强化学习,使微F1从36%提升至84%,其中13项原则的F1超过80%。该评判模型可用于审计生成界面、过滤低质训练数据,并为注重设计的代码生成提供奖励信号。我们公开数据生成方法、注入与验证提示,支持可复现评估与未来可扩展界面质量评估研究。
原文摘要 · Abstract (English)
Small language models and coding agents increasingly generate web front-end code, yet their outputs are typically evaluated primarily for functional correctness. A generated interface may compile, render, and pass unit tests while still violating established interface quality principles, including accessibility barriers, deceptive design patterns, poor visual hierarchy, and excessive decision complexity. Existing auditing approaches face a trade-off between cost, coverage, and scalability: expert human review provides rich judgment but is slow and expensive; frontier vision-language models offer broader reasoning capabilities but remain costly to deploy at scale; and rule-based tools such as axe-core and Lighthouse are inexpensive but primarily capture mechanically checkable accessibility issues. We investigate whether a lightweight vision-language model can serve as an effective critic for generated interfaces. We unify 19 interface-quality principles from three complementary sources of HCI knowledge: WCAG 2.2 accessibility standards, deceptive design taxonomies, and established theories of perception, cognition, and interaction. To train this critic, we construct a verified dataset of approximately 10,000 generated web pages by synthetically injecting known violations into clean, LLM-generated Tailwind pages. Continued reinforcement learning on a 4B vision-language model improves micro-F1 from 36\% to 84\%, with 13 of 19 principles exceeding 80\% F1. The resulting critic can audit generated interfaces, filter low-quality interface training data, and provide a reward signal for design-aware code generation. We release our data-generation recipe and injection/verification prompts to support reproducible evaluation and future work on scalable interface-quality assessment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。