提出评估代码协作中AI agents同步恢复能力的框架与基准
SyncMind: Measuring Agent Out-of-Sync Recovery in Collaborative Software Engineering
- 构建系统性框架定义LLM在协同开发中的不同步问题
- 实测显示主流LLM协作恢复率不足5%，最高仅28.18%
- 揭示现有模型缺乏资源感知力，适合研究协同智能的开发者
软件工程日益协同化，开发者需在共享复杂代码库中保持同步。当协作者（人或AI）的理解与当前环境状态偏离时，即出现‘不同步’问题，可能导致行动失败和集成问题。本文提出SyncMind框架，系统定义大型语言模型（LLM）代理在协同软件工程（CSE）中的不同步问题。基于此，我们构建了SyncBench基准，包含从21个热门GitHub仓库提取的24,332个真实世界不同步场景，配有可执行验证测试。实验揭示：现有LLM代理性能差距显著（从Llama-3.1代理≤3.33%到Claude-3.5-Sonnet≥28.18%），协作意愿极低（≤4.86%），表明其在协同场景下存在根本局限；但一旦协作，成功率与不同步恢复正相关。此外，代理在资源感知下的恢复表现差异微小，暴露其严重缺乏资源意识与自适应能力，为未来高效协同系统提供方向。代码与数据已开源：https://xhguo7.github.io/SyncMind/
原文摘要 · Abstract (English)
Software engineering (SE) is increasingly collaborative, with developers working together on shared complex codebases. Effective collaboration in shared environments requires participants -- whether humans or AI agents -- to stay on the same page as their environment evolves. When a collaborator's understanding diverges from the current state -- what we term the out-of-sync challenge -- the collaborator's actions may fail, leading to integration issues. In this work, we introduce SyncMind, a framework that systematically defines the out-of-sync problem faced by large language model (LLM) agents in collaborative software engineering (CSE). Based on SyncMind, we create SyncBench, a benchmark featuring 24,332 instances of agent out-of-sync scenarios in real-world CSE derived from 21 popular GitHub repositories with executable verification tests. Experiments on SyncBench uncover critical insights into existing LLM agents' capabilities and limitations. Besides substantial performance gaps among agents (from Llama-3.1 agent <= 3.33% to Claude-3.5-Sonnet >= 28.18%), their consistently low collaboration willingness (<= 4.86%) suggests fundamental limitations of existing LLM in CSE. However, when collaboration occurs, it positively correlates with out-of-sync recovery success. Minimal performance differences in agents' resource-aware out-of-sync recoveries further reveal their significant lack of resource awareness and adaptability, shedding light on future resource-efficient collaborative systems. Code and data are openly available on our project website: https://xhguo7.github.io/SyncMind/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。