将AI调试与评估嵌入IDE，让非机器学习背景的开发者也能高效开发AI功能。
In-IDE Toolkit for Developers of AI-Based Features
- 在IDE运行时自动捕获AI执行轨迹，支持即时层级查看。
- 一键将轨迹加入数据集，实现类似单元测试的可复现评估。
- 适合缺乏ML背景但需集成AI功能的软件工程师使用。
基于大语言模型和智能体工作流的AI功能难以测试、调试和复现，尤其对没有机器学习背景的产品开发工程师而言。我们推出了适用于JetBrains IDEs的AI Toolkit插件，将追踪与评估直接融入运行/调试流程。一项混合方法的实践研究揭示了三个核心需求：(1) 让评估常态化且可重复；(2) 在执行时刻暴露追踪信息；(3) 最小化设置和上下文切换。基于这些需求，AI Toolkit引入了原生IDE工作流：运行触发的轨迹捕获、即时分层检查、从轨迹一键“添加至数据集”，以及支持可插拔指标的类单元测试评估。首个PyCharm版本展现出积极早期信号——推广后转化率高，轨迹捕获用户持续使用，流失率低，表明将AI可观测性嵌入日常开发可显著降低使用门槛，助力开发者养成规范实践。我们详述了AI代理调试器与AI评估模块的设计与实现，报告初步采用数据，并展望扩展框架覆盖范围与规模化评估。结果表明，将AI可观测性与评估融入日常开发流程，能令非专业人员轻松应对现代AI开发，同时保持软件工程规范。
原文摘要 · Abstract (English)
AI-enabled features built on LLMs and agentic workflows are difficult to test, debug, and reproduce, especially for product-focused software engineers without a machine learning background. We present the AI Toolkit plugin for JetBrains IDEs, which brings tracing and evaluation directly into the Run/Debug loop. A mixed methods study with practitioners presents three consistent needs: (1) make evaluation regular and repeatable, (2) expose traces at the moment of execution, and (3) minimize setup and context switching. Guided by these needs, the AI Toolkit introduces an IDE-native workflow: run-triggered trace capture; immediate, hierarchical inspection; one-click "Add to Dataset" from traces; and unit-test-like evaluations with pluggable metrics. The first release in PyCharm shows promising early signals - strong conversion when promoted at Run, sustained usage among those who capture traces, and low churn - suggesting that IDE-native observability lowers activation energy and helps developers adopt disciplined practices. We detail the design and implementation of the AI Agents Debugger and AI Evaluation, report initial adoption telemetry, and outline next steps to broaden framework coverage and scale evaluations. Together, these results indicate that integrating AI observability and evaluation into everyday IDE workflows can make modern AI development accessible to non-ML specialists while preserving software-engineering practices.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。