分析了海量代码生成代理的实际使用模式,揭示其与聊天机器人的根本差异。
Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale
- 通过真实用户数据构建多步自主执行的编码代理工作负载模型
- 单轮内缓存命中率达90%,跨轮次下降至55%,模型切换后急剧失效
- 发现代理高效运行与用户长时间闲置并存,适合优化资源调度系统
以2026年6月的采样数据为基础,对生产规模的GitHub Copilot使用痕迹进行首次全面分析,涵盖320万用户、1300万会话、7.61亿次LLM调用和95万亿令牌。研究发现,代码生成代理的工作负载具有独特特征:用户发起的交互稀疏,每轮交互均触发一系列由LLM调用与工具执行组成的自主循环。单轮内部缓存命中率平均达90%,但跨轮边界降至55%,在模型切换或上下文压缩等事件后大幅失效。不同用户表现出显著差异的行为模式,包括长尾分布的令牌消耗、持续时间与工具调用次数。观察到快速的代理响应与分钟级用户空闲期并存,设计的轻量级空闲预测器可捕获86%-90%的空闲时间,为资源调度提供主动决策支持。这些发现挑战现有LLM服务系统的假设,为原生代理架构提供了实证基础。
原文摘要 · Abstract (English)
AI coding agents like GitHub Copilot, Claude Code, and Codex interleave multi-step LLM inference with tool execution, creating a workload different from chatbots. We present the first production-scale characterization of this workload using sampled GitHub Copilot traces from June 2026, comprising 3.2M users, 13M sessions, 761M LLM calls, and 95T tokens. Our analysis reveals distinctive workload properties with important systems implications. For example, agentic coding sessions consist of sparse user-initiated turns, each unfolding into an autonomous agent loop of LLM calls almost always coupled with tool execution. This structure yields KV cache hit rates averaging 90% within a turn, but falling to 55\% across turn boundaries and drastically invalidated after events like model switches or context compaction. Diverse workflows and user behaviors are observed with variable and long-tailed token consumption, time span, and tool calls. We highlight the difference between quick agentic turnaround times and the minutes-long user idle periods at turn boundaries, and design a lightweight idle-time predictor that captures 86-90\% of total idle time, enabling proactive decisions for efficient resource orchestration. These findings challenge assumptions underlying current LLM-serving systems and provide an empirical foundation for agent-native infrastructure.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。