用算法复杂度衡量提示价值,让好提示可量化。
The Value of a Prompt: An LLM-Relative Kolmogorov-Complexity Approach
- 以大模型自身为计算引擎,定义新的提示复杂度度量方式
- 提示值每增加1比特,生成目标的难度降低2倍
- 适合评估提示工程、AI创作价值与自动化研究
在由大模型生成越来越多有价值成果的时代,核心经济问题是:我们提供的输入(即提示)究竟有多大的价值?给定一个能帮助模型生成证明、程序、设计或科学假说等成果 $z$ 的提示、线索、批评或部分解答,如何衡量其价值?直观上,若提示使目标成果更易生成——提升采样概率或缩短推理时间——则具有价值。本文提出一种基于大模型相对柯尔莫哥洛夫复杂度的计算方法,将经典定义中的通用图灵机替换为大模型自身,引入“概率性莱文-柯尔莫哥洛夫复杂度” $pKt$,并以该复杂度下的算法互信息定义提示价值。该方法表明:一个价值为 $b$ 比特的提示,使目标 $z$ 的生成难度降低 $2^b$ 倍,即成功概率提升 $2^b$ 倍,或计算量减少 $2^b$ 倍。与经典算法互信息不同,本方法可高效估算。进一步实验表明,在自然复现条件下,无提示时复现 $z$ 的中位令牌成本是带提示的 $2^b$ 倍。
原文摘要 · Abstract (English)
In a world where valuable artifacts are increasingly created, completed, or processed by LLMs, the central economic question is not only what the LLM can produce, but what \emph{value} remains in the inputs (i.e., the prompts) we provide to it. Given a prompt, hint, critique, problem statement, or partial solution that helps an LLM produce an artifact $z$---a proof, program, design, or scientific hypothesis---how should we measure the value of that input? Intuitively, an input is valuable when it makes the target artifact easier for the model to generate: either by increasing its sampling probability, or by reducing the thinking time needed to find it. We propose a computational Levin--Kolmogorov complexity approach to this problem, by appropriately replacing the universal Turing machine in the classical definitions by the LLM itself. Concretely, we introduce an LLM-relative notion of \emph{probabilistic Levin--Kolmogorov complexity} $pKt$---treating the model's thinking as the random tape of the program, and charging logarithmically for it in Levin's manner---and define prompt value as algorithmic mutual information with respect to $pKt$. This captures the intuition above: a prompt having $b$ bits of value for an artifact $z$ makes $z$ $2^b$ times ``easier to obtain'', by multiplying the success probability by $2^b$, by dividing the required computation by $2^b$, or by any corresponding tradeoff between probability and computation. In contrast to the classical notion of algorithmic mutual information, ours is efficiently estimable. We additionally show that, under a natural reproduction experiment, a prompt value of \(b\) bits means that reproducing \(z\) without the prompt has median token cost \(2^b\) times that of reproducing it with the prompt.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。