构建闭环框架，让AI自动生成的GPU内核高效用于大模型推理
FlashInfer-Bench: Building the Virtuous Cycle for AI-driven LLM Systems
- 用统一格式记录内核定义与性能，实现AI与系统间通信
- 在真实服务数据上验证，支持自动替换最优内核到生产引擎
- 提供排行榜与评测工具，适合研究生成式编程与部署优化者
近期进展表明，大型语言模型（LLMs）可作为自主代理生成GPU内核，但将这些由AI生成的内核集成到实际推理系统仍具挑战。FlashInfer-Bench通过建立标准化、闭环的框架，填补了这一空白：核心是FlashInfer Trace，一种统一的描述内核定义、工作负载、实现与评估的模式，确保代理与系统间一致通信。基于真实服务追踪，该框架包含精选数据集、具备正确性与性能感知的评测系统、公开排行榜以追踪LLM代理的GPU编程能力，以及动态替换机制（apply()），可无缝将最佳性能内核注入SGLang和vLLM等生产级大模型引擎。利用此框架，我们进一步评估了LLM代理的性能与局限，对比了不同GPU编程语言的权衡，并为未来代理设计提供了洞见。FlashInfer-Bench因此建立起一条可复现、实用的路径，持续优化并部署AI生成的内核至大规模大模型推理系统。
原文摘要 · Abstract (English)
Recent advances show that large language models (LLMs) can act as autonomous agents capable of generating GPU kernels, but integrating these AI-generated kernels into real-world inference systems remains challenging. FlashInfer-Bench addresses this gap by establishing a standardized, closed-loop framework that connects kernel generation, benchmarking, and deployment. At its core, FlashInfer Trace provides a unified schema describing kernel definitions, workloads, implementations, and evaluations, enabling consistent communication between agents and systems. Built on real serving traces, FlashInfer-Bench includes a curated dataset, a robust correctness- and performance-aware benchmarking framework, a public leaderboard to track LLM agents' GPU programming capabilities, and a dynamic substitution mechanism (apply()) that seamlessly injects the best-performing kernels into production LLM engines such as SGLang and vLLM. Using FlashInfer-Bench, we further evaluate the performance and limitations of LLM agents, compare the trade-offs among different GPU programming languages, and provide insights for future agent design. FlashInfer-Bench thus establishes a practical, reproducible pathway for continuously improving AI-generated kernels and deploying them into large-scale LLM inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。