首个面向移动应用开发的缺陷修复基准，揭示大模型在此领域表现极低。
MobileDev-Bench: A Benchmark for Issue Resolution in Mobile Application Development
- 构建真实移动项目缺陷数据集，支持跨文件、多类型协同修复验证。
- 平均修复涉及12.9个文件334.6行代码，41%需跨源码、配置、资源等多类型修改。
- 四款前沿大模型端到端修复率仅3.23%-5.69%，远低于现有基准表现。
大型语言模型在自动化软件工程任务中表现出色，但现有基准主要聚焦于类库型仓库，忽视了移动应用开发的独特挑战——其框架特定的构建系统、异构的产物类型以及需多文件协同修复的需求。本文提出MobileDev-Bench，一个涵盖19个生产级移动应用的基准，覆盖Android Native（Java/Kotlin）、React Native（TypeScript）和Flutter（Dart）。该基准包含407个真实世界缺陷修复任务，每项任务配以开发者验证的缺陷报告和可执行测试补丁，支持在移动构建环境中完全自动化验证模型生成的修复方案。与已有基准相比，本基准修复复杂度显著更高：平均修复涉及12.9个文件、334.6行代码，41%的任务需要跨源码、构建配置、资源文件等多类型协同修改。对四种前沿大模型（Claude Sonnet 4.5、Qwen3-Coder、GPT-5.2、Gemini 2.5 Flash）的评估显示，自动检索下端到端修复率仅为3.23%–4.23%，即便使用理想检索（oracle retrieval）也仅达5.69%，远低于现有基准报告结果。我们开源MobileDev-Bench，包含任务实例、评估工具链及容器化环境，以支持可复现的AI辅助移动开发研究。
原文摘要 · Abstract (English)
Large language models (LLMs) have shown strong performance on automated software engineering tasks, yet existing benchmarks focus primarily on library-style repositories, leaving mobile application development largely unexplored despite its framework-specific build systems, heterogeneous artifact types, and coordinated multi-file fix requirements. We introduce MobileDev-Bench, a benchmark comprising 407 real-world issue-resolution tasks collected from 19 production mobile applications spanning Android Native (Java/Kotlin), React Native (TypeScript), and Flutter (Dart). Each task pairs a verified developer-reported issue with executable test patches, enabling fully automated validation of model-generated fixes within mobile build environments. The benchmark exhibits substantially greater patch complexity than prior benchmarks: fixes modify 12.9 files and 334.6 lines on average, and 41% of instances require coordinated changes across multiple artifact types, such as source, build configuration, and resource files. Evaluation of four frontier LLMs (Claude Sonnet 4.5, Qwen3-Coder, GPT-5.2, and Gemini 2.5 Flash) yields end-to-end resolution rates of only 3.23% - 4.23% under automated retrieval and at most 5.69% under oracle retrieval, well below resolution rates reported on existing benchmarks. We release MobileDev-Bench with task instances, an evaluation harness, and containerized environments to support reproducible research on AI-assisted mobile application development.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。