发现多模态模型生成代码时会受重复视觉模式干扰，导致错误输出。
Pattern over Pixels: Measuring Pattern Completion Bias in Multimodal Code Generation
- 构建首个视觉模式补全偏差评测基准，测试模型对局部扰动的反应。
- 平均偏差率高达80.22%，准确率仅7.89%，噪声和边界位置加剧问题。
- 模型能识别异常但仍优先选模式一致答案，适合关注视觉误导的研究者。
多模态大语言模型（MLLMs）被广泛用于将网页截图转换为前端代码，但重复的用户界面（UI）模式可能导致模型生成看似合理却在视觉上不正确的代码。本文通过一个客观的截图-代码填空任务，测试重复模式对模型准确性的影响。我们构建了首个针对视觉模式补全偏差的基准，其中在重复的UI模式中局部元素被扰动，模型需根据截图和HTML上下文恢复被遮蔽的宽度或字体大小。基于Design2Code数据集中的30个网页，生成1,440张评测截图，涵盖结构型卡片与文本样式模式，在标准和叠加噪声条件下进行测试。评估五个前沿的MLLMs，发现所有模型均强烈偏向重复基线：卡片宽度扰动下平均偏差率达69.78%，准确率仅21.17%；文本字体大小扰动下偏差率达80.22%，准确率仅为7.89%。Codex-5.3表现最佳，但在文本任务上准确率从68.61%降至13.89%；Flash-3.0在文本任务上偏差率达96.11%。噪声、微小扰动及边界位置进一步提升偏差率。推理分析显示，更复杂的推理与更低偏差相关，但定性证据表明模型虽能识别异常元素，仍会强制输出模式一致的结果。研究揭示了多模态代码生成中的具体失败模式，其严重程度与视觉显著性密切相关。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) are increasingly used to translate webpage screenshots into front-end code, but repeated UI patterns may sway them toward visually incorrect yet pattern-consistent outputs. In this work, we test how repeated webpage patterns hurt MLLM accuracy on an objective screenshot-to-code fill-in-the-blank task. We introduce the first benchmark for visual pattern-completion bias, where one localized element in a repeated UI pattern is perturbed and the model must recover the masked width or font-size value from the screenshot and HTML context. Starting from 30 webpages curated from the Design2Code dataset, we build 1,440 evaluated screenshots spanning structural card and text-style patterns under standard and noise-overlaid conditions. We evaluate five frontier MLLMs and find that all are strongly biased toward the repeated baseline. Mean bias rate reaches 69.78% on card-width perturbations and 80.22% on text font-size perturbations, while mean accuracy is only 21.17% and 7.89%, respectively. Codex-5.3 performs best but still drops from 68.61% accuracy on cards to 13.89% on text, while Flash-3.0 reaches 96.11% bias on text. Noise, subtler perturbations, and boundary positions further increase bias rate. Reasoning analysis further shows that greater reasoning effort correlates with lower bias, yet qualitative evidence reveals that models can identify the anomalous element and still override it with the pattern-consistent answer. Our results identify a concrete failure mode in multimodal code generation and show that its severity is strongly associated with visual saliency
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。