首次利用视觉数据提升代码问题自动修复能力
CodeV: Issue Resolving with Visual Data
- 通过双阶段流程处理视觉与文本数据
- 在Visual SWE-bench上显著优于仅用文本的方法
- 适合关注工程效率与多模态分析的研究者
近年来，大语言模型（LLMs）在软件工程领域快速发展，其应用已扩展至更复杂的仓库级任务。GitHub问题修复是其中的关键挑战。现有方法虽有进展，但仅依赖问题中的文本数据，忽视了对问题解决至关重要的视觉信息。实际上，视觉数据能传递文本无法表达的额外知识。为此，我们提出CodeV，首个利用视觉数据增强LLM问题修复能力的方法。CodeV采用两阶段流程：数据处理与补丁生成。为评估CodeV，我们构建了首个面向视觉问题修复的基准测试集Visual SWE-bench。大量实验表明，CodeV有效提升了修复性能，并揭示了视觉数据在问题解决中的关键作用。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have advanced rapidly in recent years, with their applications in software engineering expanding to more complex repository-level tasks. GitHub issue resolving is a key challenge among these tasks. While recent approaches have made progress on this task, they focus on textual data within issues, neglecting visual data. However, this visual data is crucial for resolving issues as it conveys additional knowledge that text alone cannot. We propose CodeV, the first approach to leveraging visual data to enhance the issue-resolving capabilities of LLMs. CodeV resolves each issue by following a two-phase process: data processing and patch generation. To evaluate CodeV, we construct a benchmark for visual issue resolving, namely Visual SWE-bench. Through extensive experiments, we demonstrate the effectiveness of CodeV, as well as provide valuable insights into leveraging visual data to resolve GitHub issues.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。