用大模型提示词辅助软件性能配置，省去大量试错。
Prompting for Performance: Exploring LLMs for Configuring Software
- 用提示词让大模型识别关键配置项，替代人工经验。
- 在编译器、视频编码器等系统中，推荐效果接近专家水平。
- 适合需要快速调优但缺乏领域知识的开发者使用。
软件系统通常包含大量影响执行时间、内存占用、二进制大小或码率等性能指标的配置选项。一方面，做出明智决策需深厚的领域知识；另一方面，机器学习虽能搜索广阔配置空间，但需大量实际运行，计算成本高昂。本探索性研究考察大语言模型（LLMs）通过提示词辅助性能导向的软件配置的可行性。我们在编译器、视频编码器、SAT求解器等多种可配置系统上评估多个LLM在识别相关选项、排序配置和推荐高性能配置任务中的表现。初步结果表明，根据任务和系统不同，LLMs能较好契合专家判断，但也存在幻觉或浅层推理等问题。这些发现标志着系统评估与基于大模型的配置解决方案设计的第一步。
原文摘要 · Abstract (English)
Software systems usually provide numerous configuration options that can affect performance metrics such as execution time, memory usage, binary size, or bitrate. On the one hand, making informed decisions is challenging and requires domain expertise in options and their combinations. On the other hand, machine learning techniques can search vast configuration spaces, but with a high computational cost, since concrete executions of numerous configurations are required. In this exploratory study, we investigate whether large language models (LLMs) can assist in performance-oriented software configuration through prompts. We evaluate several LLMs on tasks including identifying relevant options, ranking configurations, and recommending performant configurations across various configurable systems, such as compilers, video encoders, and SAT solvers. Our preliminary results reveal both positive abilities and notable limitations: depending on the task and systems, LLMs can well align with expert knowledge, whereas hallucinations or superficial reasoning can emerge in other cases. These findings represent a first step toward systematic evaluations and the design of LLM-based solutions to assist with software configuration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。