用大模型生成编译器优化流程,显著提升长尾任务性能。
PassNet: Scaling Large Language Models for Graph Compiler Pass Generation

- 提出基于大模型的编译器优化流程生成框架PassNet,支持直接集成到编译管线中。
- 在2060个子图上实测,大模型单个子图最高提速3倍,整体仍落后主流编译器37%。
- 提供18000+真实模型图和200个长尾任务基准,适合编译器与AI交叉研究者使用。
现代张量编译器如TorchInductor在主流模型上实现显著加速,但在长尾工作负载上面临系统性性能瓶颈——我们的分析显示,43%的真实子图在默认编译下出现端到端性能下降。尽管大语言模型(LLM)为自动化优化提供了可能,现有工作集中于独立核函数生成。我们主张更合适的抽象是“编译器优化流程生成”(pass generation),即由大模型生成可直接嵌入编译器流水线的结构化图变换。为此,我们提出PassNet,首个大规模面向大模型的编译器优化流程生成生态,包含:(1) PassNet-Dataset,来自10万份真实模型的超过18,000个独特计算图;(2) PassBench,200个精心筛选的长尾融合任务(共2,060个子图),采用误差感知加速分数(ES_t)评估,该指标统一考量正确性、稳定性与性能,并设有分层完整性防御以应对大模型系统性滥用。实验表明,PassBench具有高度区分性和未饱和特性:最优前沿模型整体仍比TorchInductor慢37%,但在个别子图上大模型可达3倍加速,说明瓶颈在于一致性而非能力。仅用约4,000条PassNet轨迹微调小型模型,即可实现2.67倍性能提升,接近前沿模型水平,证明了其巨大的优化空间并验证了PassNet作为实时训练基础设施的有效性。所有数据、基准与工具均公开可用。
原文摘要 · Abstract (English)
Modern tensor compilers such as TorchInductor deliver substantial speedups on mainstream models, yet face a systematic performance ceiling on long-tail workloads -- our profiling shows that 43% of real-world subgraphs experience end-to-end slowdowns under default compilation. While LLMs offer a path toward automated optimization, existing efforts focus on standalone kernel generation. We argue that pass generation -- where LLMs author structured graph transformations that integrate directly into compiler pipelines -- is the more appropriate abstraction. We propose PassNet, the first large-scale ecosystem for LLM-based compiler pass generation, comprising: (1) PassNet-Dataset, over 18K unique computational graphs from 100K real-world models; and (2) PassBench, 200 curated long-tail fusible tasks (comprising 2,060 subgraphs in total) evaluated under the Error-aware Speedup Score (ES_t) -- a metric unifying correctness, stability, and performance -- with layered integrity defenses against systematic LLM exploitation. Experiments reveal that PassBench is both highly discriminative and genuinely unsaturated: the best frontier model trails TorchInductor by 37% in aggregate, yet on individual subgraphs LLMs achieve up to 3x speedup over the same compiler -- indicating that the bottleneck is consistency, not capability. Fine-tuning a small model on merely ~4K PassNet trajectories yields a 2.67x improvement approaching frontier-model performance, demonstrating substantial headroom and validating PassNet as live training infrastructure for advancing LLM-driven compiler optimization. All data, benchmarks, and tooling are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。