用自动分析预测LLM对提示词的反应，省时高效。
Predictive Prompt Analysis
- 基于稀疏自编码器构建提示词预测模型
- 预测代码生成中语法结构出现频率相关性达0.994
- 仅需LLM运行时间的0.4%，适合开发者提速
大语言模型（LLMs）因处理复杂任务的能力而广泛应用，但其表现高度依赖提示词设计。当前优化提示词主要靠试错，成本高且缺乏规律可循。本文提出‘预测式提示分析’概念，即通过自动化方法快速评估提示词并预测模型响应。以语法普遍性分析器（SPA）为例，该方法基于稀疏自编码器（SAEs），能准确预测大语言模型在代码生成中目标语法结构的出现频率，预测值与实际值的皮尔逊相关系数高达0.994。同时，SPA仅需原任务计算时间的0.4%。随着LLM在软件开发中的深度集成，该方法有望显著降低使用门槛，提升研究与实践效率。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are machine learning models that have seen widespread adoption due to their capability of handling previously difficult tasks. LLMs, due to their training, are sensitive to how exactly a question is presented, also known as prompting. However, prompting well is challenging, as it has been difficult to uncover principles behind prompting -- generally, trial-and-error is the most common way of improving prompts, despite its significant computational cost. In this context, we argue it would be useful to perform `predictive prompt analysis', in which an automated technique would perform a quick analysis of a prompt and predict how the LLM would react to it, relative to a goal provided by the user. As a demonstration of the concept, we present Syntactic Prevalence Analyzer (SPA), a predictive prompt analysis approach based on sparse autoencoders (SAEs). SPA accurately predicted how often an LLM would generate target syntactic structures during code synthesis, with up to 0.994 Pearson correlation between the predicted and actual prevalence of the target structure. At the same time, SPA requires only 0.4\% of the time it takes to run the LLM on a benchmark. As LLMs are increasingly used during and integrated into modern software development, our proposed predictive prompt analysis concept has the potential to significantly ease the use of LLMs for both practitioners and researchers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。