用概率编程语言让大模型流水线可量化不确定性。
PPDL: LLM-Based Flows as Probabilistic Programs
- 将大模型调用流程建模为概率程序，自动追踪不确定度
- 支持不同推理策略实验，无需修改流程逻辑代码
- 适合需要可信输出的AI系统开发者使用
构建可靠的大型语言模型（LLM）应用仍面临重大挑战。尽管LLM在各类任务中表现优异，但其输出常缺乏准确性，且无明确置信度衡量。多步调用LLM及其他工具的流程中，不确定性会累积，使开发者与用户难以信任结果。本文提出一种用于编程LLM流的概率语言。该语言使开发者能够量化并传播整个应用流程中的不确定性，并在不额外添加代码的前提下，试验不同的推理缩放技术。我们通过实验研究验证了该能力，并以构建Rocq定理证明代理为例进行了案例研究。
原文摘要 · Abstract (English)
Building reliable applications that leverage large language models (LLMs) remains a significant challenge. While LLMs offer impressive capabilities across diverse tasks, their outputs often lack accuracy and provide no clear measure of confidence. This uncertainty compounds in flows of multiple calls to LLMs and other tools, making it difficult for developers and end-users to trust the results. This paper introduces a probabilistic language for programming LLM-based flows. It enables developers to quantify and propagate uncertainty throughout the application's flow, and experiment with different inference scaling techniques without adding a single line of code beyond the flow's logic. We present an experimental study to demonstrate this capability, and a case study building a theorem proving agent for the Rocq theorem prover.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。