自动化大模型应用全生命周期管理，提升质量并缩短发布周期
Generative AI Toolkit -- a framework for increasing the quality of LLM-based applications over their whole life cycle
- 构建全流程自动化工具链，覆盖配置、测试、监控与优化
- 显著提升大模型应用质量，缩短发布周期，支持持续迭代
- 适合需要高效维护大模型应用的开发与运维团队
随着基于大语言模型（LLM）的应用服务数百万用户，确保其可扩展性和持续质量改进至关重要。然而，当前开发、维护和运营（DevOps）这些应用的流程仍以人工为主，效率低且依赖试错。本文提出 Generative AI Toolkit，实现大模型应用全生命周期关键流程的自动化。该工具链支持代理类应用的配置、测试、持续监控与优化，显著提升应用质量并加速发布周期。我们在典型应用场景中验证了其有效性，分享最佳实践，并展望未来改进方向。我们坚信该工具对其他团队有帮助，因此开源发布，欢迎使用、传播、改进与扩展。
原文摘要 · Abstract (English)
As LLM-based applications reach millions of customers, ensuring their scalability and continuous quality improvement is critical for success. However, the current workflows for developing, maintaining, and operating (DevOps) these applications are predominantly manual, slow, and based on trial-and-error. With this paper we introduce the Generative AI Toolkit, which automates essential workflows over the whole life cycle of LLM-based applications. The toolkit helps to configure, test, continuously monitor and optimize Generative AI applications such as agents, thus significantly improving quality while shortening release cycles. We showcase the effectiveness of our toolkit on representative use cases, share best practices, and outline future enhancements. Since we are convinced that our Generative AI Toolkit is helpful for other teams, we are open sourcing it on and hope that others will use, forward, adapt and improve
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。