真实开发者行为数据揭示了代码助手模拟评估的严重偏差。
An Empirical Study of Proactive Coding Assistants in Real-World Software Development

- 用真实IDE操作数据对比仿真数据,发现行为模式差异显著
- 实测显示当前智能助手在真实场景下性能远低于仿真结果
- 提出首个真实世界主动意图预测基准,强调真实数据的重要性
基于大语言模型(LLM)的代码助手机能已取得显著进展,但多数系统仍为被动响应式,需开发者明确表达需求。主动式代码助手则试图从IDE交互与代码库上下文推断开发者潜在意图,以降低交互成本并实现更自然的辅助。然而,该方向研究受限于大规模真实开发者行为数据的缺乏,现有研究多依赖LLM生成的仿真IDE轨迹,其与真实开发行为的吻合度尚不明确。本文通过一项大规模实证研究,采集了1,246名资深产业开发者连续三天的真实IDE操作日志,并构建对应的LLM仿真轨迹进行对照分析。结果表明,仿真轨迹在行为多样性、时间结构和探索模式上与真实轨迹存在显著差异。基于此数据,我们提出了首个真实世界主动意图预测基准——ProCodeBench。对代表性LLM、检索增强方法及代理基线的实验显示,当前方法在真实轨迹下的表现远未可靠,说明基于仿真的评估可能严重高估实际性能。最后,训练研究表明,仿真数据无法替代真实数据,但在真实微调前可作为补充。这些发现凸显了真实开发者行为数据在评估与训练主动代码助手中的关键作用。
原文摘要 · Abstract (English)
Large language model (LLM)-based coding assistants have made substantial progress, yet most systems remain reactive, requiring developers to explicitly formulate their needs. Proactive coding assistants aim to infer latent developer intent from integrated development environment (IDE) interactions and repository context, thereby reducing interaction overhead and supporting more seamless assistance. However, research in this direction is limited by the scarcity of large-scale real-world developer behavior data. Existing studies therefore often rely on LLM-simulated IDE traces, whose fidelity to real development behavior remains unclear. In this paper, we investigate this simulation-to-reality gap through a large-scale empirical study. We collect real IDE interaction traces from 1{,}246 experienced industry developers over three consecutive days using a custom Visual Studio Code extension, and construct paired LLM-simulated traces for controlled comparison. Our analysis shows that simulated traces differ substantially from real traces in behavioral diversity, temporal structure, and exploratory patterns. Based on the collected data, we introduce \textbf{ProCodeBench}, a real-world benchmark for proactive intent prediction. Experiments with representative LLMs, retrieval-augmented methods, and agentic baselines show that current approaches remain far from reliable under real IDE traces, suggesting that simulation-based evaluation can overestimate real-world performance. Finally, our training study shows that simulated data cannot replace real data, but can complement it when used before real-world fine-tuning. These findings highlight the importance of real developer behavior data for evaluating and training proactive coding assistants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。