用大模型发现优化，再转为通用规则，提升Pandas程序速度4.3倍。
RuleFlow : Generating Reusable Program Optimizations with LLMs
- 先用大模型发现具体优化，再转为可复用的通用重写规则
- 在PandasBench上最快提速4.3倍，比之前最优快1914.9倍
- 适合需要高效运行Pandas代码的开发者和系统设计者
优化Pandas程序是一项挑战。现有系统和基于编译器的方法虽可靠，但或过于复杂，或仅支持有限优化；而直接使用大模型进行逐程序优化虽能生成复杂优化，却不可靠、成本高且效率低。本文提出一种三阶段混合方法：首先发现程序级优化（发现）；其次将优化转化为通用重写规则（桥接）；最后将规则集成到编译器中自动应用，避免重复调用大模型（部署）。在包含Python笔记本的PandasBench基准测试中，RuleFlow成为新的最先进框架，相比前代编译器最优方案Dias实现最高4.3倍加速，相比系统级最优方案Modin实现高达1914.9倍加速。代码已开源于https://github.com/ADAPT-uiuc/RuleFlow。
原文摘要 · Abstract (English)
Optimizing Pandas programs is a challenging problem. Existing systems and compiler-based approaches offer reliability but are either heavyweight or support only a limited set of optimizations. Conversely, using LLMs in a per-program optimization methodology can synthesize nontrivial optimizations, but is unreliable, expensive, and offers a low yield. In this work, we introduce a hybrid approach that works in a 3-stage manner that decouples discovery from deployment and connects them via a novel bridge. First, it discovers per-program optimizations (discovery). Second, they are converted into generalised rewrite rules (bridge). Finally, these rules are incorporated into a compiler that can automatically apply them wherever applicable, eliminating repeated reliance on LLMs (deployment). We demonstrate that RuleFlow is the new state-of-the-art (SOTA) Pandas optimization framework on PandasBench, a challenging Pandas benchmark consisting of Python notebooks. Across these notebooks, we achieve a speedup of up to 4.3x over Dias, the previous compiler-based SOTA, and 1914.9x over Modin, the previous systems-based SOTA. Our code is available at https://github.com/ADAPT-uiuc/RuleFlow.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。