arXiv:2510.07356cs.LGcs.CL2025-10被引 13

用简洁推理生成高质量CUDA内核,性能超越现有顶尖模型。

ConCuR: Conciseness Makes State-of-the-Art Kernel Generation

  • 通过简洁且信息丰富的推理过程自动生成并筛选高质内核数据。
  • 在KernelBench上优于QwQ-32B、DeepSeek-V3.1-Think等前沿模型。
  • 提出推理长度可衡量任务难度,助力未来数据构建。

基于大语言模型(LLM)的GPU内核生成近年来发展迅速,借助测试时扩展和强化学习技术。然而,内核生成的关键挑战在于高质量数据稀缺,因多数优质内核为专有代码且未开源,导致难以使用监督微调对齐模型。为此,我们设计了一条生成与筛选高质量CUDA内核的流水线,其核心观察是:简洁而富有信息量的推理轨迹能有效生成高性能内核。基于该流水线,我们构建了数据集ConCuR,并提出首个在包含PyTorch、推理轨迹与CUDA内核三元组的精选数据集上训练的模型KernelCoder。在KernelBench评测中,该模型显著优于当前最优模型QwQ-32B,同时超越所有针对内核生成微调的开源模型,以及DeepSeek-V3.1-Think和Claude-4-sonnet等前沿模型。最后,我们验证平均推理长度可作为评估内核生成任务难度的指标。这些发现、度量方法及数据构建流程将有助于未来获取更优数据。

原文摘要 · Abstract (English)

GPU kernel generation by LLMs has recently experienced rapid development, leveraging test-time scaling and reinforcement learning techniques. However, a key challenge for kernel generation is the scarcity of high-quality data, as most high-quality kernels are proprietary and not open-source. This challenge prevents us from leveraging supervised fine-tuning to align LLMs to the kernel generation task. To address this challenge, we develop a pipeline that generates and curates high-quality CUDA kernels with reasoning traces, motivated by a critical observation that concise yet informative reasoning traces result in robust generation of high-performance kernels. Using this pipeline, we construct our dataset ConCuR and introduce our model KernelCoder, which is the first model trained on a curated dataset consisting of PyTorch, reasoning, and CUDA kernel pairs, to our knowledge. In the KernelBench setup, our model achieves significant improvements over the existing top-performing model, QwQ-32B, and outperforms all open-source models fine-tuned for kernel generation, as well as frontier models such as DeepSeek-V3.1-Think and Claude-4-sonnet. Finally, we show that the average reasoning length can serve as a metric to assess the difficulty of kernel generation tasks. The observations, metrics, and our data collection and curation pipeline can help obtain better data in the kernel generation task in the future.

GPU内核LLM生成数据构建

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。