首个真实开发者使用AI编程助手的大规模数据集,揭示其实际效果与局限。
SWE-chat: Coding Agent Interactions From Real Users in the Wild

- 从开源项目中自动收集6000个真实编码会话,记录用户提问与代理调用。
- 仅44%的代理生成代码被提交,且漏洞比人工编写更多。
- 适合研究AI编程助手在真实开发流程中的表现与优化方向。
AI编程助手正大规模应用,但缺乏对真实使用情况和产出有效性的实证研究。本文提出SWE-chat,首个从真实开源开发者中收集的大规模编码代理会话数据集。当前包含6000个会话,超过6.3万条用户提示和35.5万次代理工具调用。该数据集持续更新,可自动发现并处理公共仓库中的会话。基于SWE-chat,我们初步刻画了真实世界中编码代理的使用模式与失效情形:编码行为呈双峰分布——41%会话中代理撰写几乎全部提交代码(“氛围编程”),23%会话中人类全程自主编码。尽管能力快速提升,代理在自然环境中仍效率低下:仅44%的代理生成代码进入用户提交,且引入的漏洞多于人工代码。用户在44%的交互回合中通过修正、报告失败或中断来抵制代理输出。通过完整记录交互轨迹并标注人类与代理的代码归属,SWE-chat为超越精心设计基准、建立基于证据的开发流程中AI代理性能理解提供了实证基础。
原文摘要 · Abstract (English)
AI coding agents are being adopted at scale, yet we lack empirical evidence on how people actually use them and how much of their output is useful in practice. We present SWE-chat, the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild. The dataset currently contains 6,000 sessions, comprising more than 63,000 user prompts and 355,000 agent tool calls. SWE-chat is a living dataset; our collection pipeline automatically and continually discovers and processes sessions from public repositories. Leveraging SWE-chat, we provide an initial empirical characterization of real-world coding agent usage and failure modes. We find that coding patterns are bimodal: in 41% of sessions, agents author virtually all committed code ("vibe coding"), while in 23%, humans write all code themselves. Despite rapidly improving capabilities, coding agents remain inefficient in natural settings. Just 44% of all agent-produced code survives into user commits, and agent-written code introduces more security vulnerabilities than code authored by humans. Furthermore, users push back against agent outputs -- through corrections, failure reports, and interruptions -- in 44% of all turns. By capturing complete interaction traces with human vs. agent code authorship attribution, SWE-chat provides an empirical foundation for moving beyond curated benchmarks towards an evidence-based understanding of how AI agents perform in real developer workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。