用视觉结构图提升程序修复的精准度，让AI更懂界面错误。
SVRepair: Structured Visual Reasoning for Automated Program Repair
- 将截图、流程图等视觉信息转为语义场景图，统一表达界面元素关系
- 在SWE-Bench上修复35.98%的实例，视觉类任务修复率21.05%
- 适合做界面相关程序错误修复的研究者和开发者
大语言模型（LLMs）被用于自动化程序修复（APR），但多数方法仍为单模态，未能利用截图和控制流图中的诊断信号。实际中，许多错误报告通过视觉方式传递关键信息（如布局错乱或控件缺失），但直接输入密集视觉数据常导致上下文丢失和噪声，使多模态大模型难以准确定位故障并生成可执行补丁。为此，我们提出SVRepair，一种基于结构化视觉表示（SVR）的多模态APR框架。该框架首先微调一个视觉-语言模型，将异构视觉要素统一转换为包含界面元素及其结构关系（如层级）的语义场景图，为下游修复提供标准化、与代码相关的上下文。在此基础上，SVRepair驱动编码代理进行故障定位与补丁生成，并引入迭代视觉区域分割策略，逐步聚焦于与错误相关的区域，抑制无关上下文并减少幻觉。在主流仓库级APR基准测试中，SVRepair成功修复186/517个SWE-Bench M实例（占全部实例的35.98%，提交运行的36.47%），以及4/19个视觉类OmniGIRL实例（21.05%）。在补充的结构化多模态代码推理基准上，其表现达MMCode 38.02% 和 CodeVision 95.73%。代码已开源：https://github.com/codefuse-ai/CodeFuse-SVR。
原文摘要 · Abstract (English)
Large language models (LLMs) have recently been applied to Automated Program Repair (APR), yet most existing approaches remain unimodal and fail to use diagnostic signals contained in visual artifacts such as screenshots and control-flow graphs. In practice, many bug reports convey critical information visually (e.g., layout breakage or missing widgets), but directly using such dense visual inputs often causes context loss and noise, making it difficult for MLLMs to ground visual observations into precise fault localization and executable patches. To bridge this semantic gap, we propose \textbf{SVRepair}, a multimodal APR framework with Structured Visual Representation (SVR). SVRepair first fine-tunes a vision-language model, SVR, to uniformly transform heterogeneous visual artifacts into a \emph{semantic scene graph} that captures GUI elements and their structural relations (e.g., hierarchy), providing normalized, code-relevant context for downstream repair. Building on the graph, SVRepair drives a coding agent to localize faults and synthesize patches, and further introduces an iterative visual-artifact segmentation strategy that progressively narrows the input to bug-centered regions to suppress irrelevant context and reduce hallucinations. Across primary repository-level APR benchmarks, SVRepair resolves \textbf{186/517} SWE-Bench M instances (\textbf{35.98\%} over all instances; \textbf{36.47\%} over submitted runs) and \textbf{4/19} visual OmniGIRL instances (\textbf{21.05\%}). On supplementary structured multimodal code reasoning benchmarks, SVRepair reaches \textbf{38.02\%} on MMCode and \textbf{95.73\%} on CodeVision. Code is available at https://github.com/codefuse-ai/CodeFuse-SVR.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。