用大模型自动检测代码阶段的用户体验缺陷，效果可靠但严重性判断需人工核验。
Catching UX Flaws in Code: Leveraging LLMs to Identify Usability Flaws at the Development Stage
- 用GPT-4o对30个开源网站执行十项可用性启发式评估，每站独立评三次。
- 问题识别一致性中等（平均Cohen's Kappa=0.50），准确率达84%；严重性判断差异大。
- 适合早期开发阶段快速筛查问题，但严重性评估仍需人类把关。
可用性评估对确保现代界面满足用户需求至关重要，但传统由人工专家进行的启发式评估耗时且主观，尤其在开发初期。本文探究大型语言模型（LLMs）是否能在开发阶段提供可靠、一致的启发式评估。通过将雅各布·尼尔森的十条可用性启发式应用于30个开源网站，我们使用OpenAI的GPT-4o生成了超过850次启发式评估，每个网站进行三次独立评估。在问题检测方面，模型表现出中等一致性，平均成对Cohen's Kappa为0.50，精确一致率达84%。严重性判断则波动更大：加权Cohen's Kappa平均为0.63，但精确一致率仅为56%，Krippendorff's Alpha接近零。结果表明，尽管GPT-4o能生成内部一致的评估，特别是在识别可用性问题存在性方面表现良好，但其严重性判断能力不稳定，实际应用中仍需人类监督。本研究揭示了利用大模型进行早期自动化可用性测试的可行性与局限性，并为提升自动化用户体验评估的一致性提供了基础。据我们所知，这是首个对自动化启发式评估进行量化评分者间可靠性分析的工作，指出了改进模型一致性的方法。
原文摘要 · Abstract (English)
Usability evaluations are essential for ensuring that modern interfaces meet user needs, yet traditional heuristic evaluations by human experts can be time-consuming and subjective, especially early in development. This paper investigates whether large language models (LLMs) can provide reliable and consistent heuristic assessments at the development stage. By applying Jakob Nielsen's ten usability heuristics to thirty open-source websites, we generated over 850 heuristic evaluations in three independent evaluations per site using a pipeline of OpenAI's GPT-4o. For issue detection, the model demonstrated moderate consistency, with an average pairwise Cohen's Kappa of 0.50 and an exact agreement of 84%. Severity judgments showed more variability: weighted Cohen's Kappa averaged 0.63, but exact agreement was just 56%, and Krippendorff's Alpha was near zero. These results suggest that while GPT-4o can produce internally consistent evaluations, especially for identifying the presence of usability issues, its ability to judge severity varies and requires human oversight in practice. Our findings highlight the feasibility and limitations of using LLMs for early-stage, automated usability testing, and offer a foundation for improving consistency in automated User Experience (UX) evaluation. To the best of our knowledge, our work provides one of the first quantitative inter-rater reliability analyses of automated heuristic evaluation and highlights methods for improving model consistency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。