为网页界面视觉变更检测提供自然语言描述的基准数据集。
Beyond Pixel Diffs: Benchmarking Image Change Captioning for Web UI Visual Regression Testing
- 提出网页界面变化字幕生成新任务,用自然语言描述界面改动。
- 构建首个公开数据集WUICC-bench,支持对细微布局和文字变化的精准描述。
- 验证现有模型在真实网页场景下比像素对比更少误报,适合测试团队使用。
视觉回归测试(VRT)是现代软件发布流程中的标准质量保障步骤。每次更新后,系统会重新渲染用户界面截图,与批准的基线图像进行对比,并将检测到的差异转交人工审核,判断是否为有意修改或意外缺陷。目前广泛采用的像素级比较方法语义盲区大,将渲染噪声与真实缺陷同等对待,导致大量误报,迫使开发和测试人员在每次发布周期中投入大量时间手动审查。行业工具虽引入机器学习,但缺乏公开评估。更关键的是,尚无公开数据集或基准支持对界面变化的自然语言描述——这种能力能以文字形式告知测试者具体发生了什么,而非仅依赖二值标记或高亮区域。为此,我们提出新任务:网页界面图像变化字幕生成(WUICC),融合视觉回归测试与图像差异描述,发布首个数据集和基准WUICC-bench。我们评估了11种代表性图像差异描述方法及2个零样本通用大模型,发现:(1)这些方法在网页界面领域表现受限,因布局多样、文本密集、变化细微;(2)但训练过的模型已显著比像素级对比更精准地过滤非语义噪声,为未来特定领域研究奠定坚实基础。
原文摘要 · Abstract (English)
Visual regression testing (VRT) is a standard quality assurance step in modern software release pipelines. On every change, it re-renders user interface (UI) screenshots, compares each one against an approved baseline image, and routes any detected difference to a human reviewer who decides whether it is an intended update or an unintended regression. A widely used approach, especially in open-source and continuous-integration pipelines, is pixel-level comparison, which is semantically blind and treats rendering noise and genuine defects identically, producing large volumes of false positives that force developers and testers to spend substantial time and effort manually reviewing flagged differences at every release cycle. Industry tools apply machine learning to VRT, but lack public evaluation. More critically, no dataset or benchmark exists to support natural language descriptions of UI changes, a capability that tells testers what changed in words instead of leaving them to interpret a binary flag or a highlighted region. To address the gap, we propose a new task, Web UI Image Change Captioning (WUICC), which sits at the intersection of VRT and image difference captioning (IDC), and release WUICC-bench, its first dataset and benchmark for the task. We evaluate eleven representative IDC methods, together with two zero-shot general-purpose LLMs. We find that: (1) these methods tend to struggle in the Web UI domain due to its layout diversity, dense text, and fine-grained changes, and (2) yet the trained methods already suppress non-meaningful visual noise far more selectively than the pixel-level comparison VRT relies on, providing a solid foundation for future domain-specific research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。