大模型靠推理就能学会简单代码优化，思维链是关键。
Towards LLM-based optimization compilers. Can LLMs learn how to apply a single peephole optimization? Reasoning is all LLMs need!
- 用思维链推理让大模型学会应用单一代码优化
- 未微调的GPT-o1比微调的Llama2更优
- 适合研究代码生成与优化的学者参考
大型语言模型（LLMs）在自然语言处理中展现出巨大潜力，近期研究探索其在编译器优化中的应用。然而，现有工作多基于Llama2等开源模型，缺乏强化推理能力。本文研究微调后的70亿参数Llama2模型在学习和应用AArch64汇编代码单一窥视优化时产生的错误，并与具备先进推理逻辑的OpenAI模型（包括GPT-4o和GPT-o1预览版）进行对比。结果表明，尽管未经过微调，GPT-o1的表现优于微调后的Llama2和GPT-4o。这一优势主要归因于GPT-o1内置的思维链推理机制。研究提示，具备增强推理能力的大模型在代码生成与优化中具有广阔前景。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated great potential in various language processing tasks, and recent studies have explored their application in compiler optimizations. However, all these studies focus on the conventional open-source LLMs, such as Llama2, which lack enhanced reasoning mechanisms. In this study, we investigate the errors produced by the fine-tuned 7B-parameter Llama2 model as it attempts to learn and apply a simple peephole optimization for the AArch64 assembly code. We provide an analysis of the errors produced by the LLM and compare it with state-of-the-art OpenAI models which implement advanced reasoning logic, including GPT-4o and GPT-o1 (preview). We demonstrate that OpenAI GPT-o1, despite not being fine-tuned, outperforms the fine-tuned Llama2 and GPT-4o. Our findings indicate that this advantage is largely due to the chain-of-thought reasoning implemented in GPT-o1. We hope our work will inspire further research on using LLMs with enhanced reasoning mechanisms and chain-of-thought for code generation and optimization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。