用大模型做概率推理，避免生成顺序偏差。
Structured Inference with Large Language Gibbs
- 用大模型的逐词条件分布作为马尔可夫链采样算子
- 在合成分布和贝叶斯结构学习中表现稳定，收敛到一致分布
- 适合需要多轮逻辑协调的任务，如复杂推理与结构推断
大语言模型（LLM）蕴含的知识可作为复杂世界变量结构化推理的基础，但如何以概率一致的方式访问这些知识仍是一个难题。本文提出大型语言吉布斯（Large Language Gibbs），一种基于LLM条件分布作为转移算子的结构化概率推理方法。不同于单次自回归生成结构化对象，该方法通过迭代地根据其他变量条件重采样单个变量，利用LLM的下一个词条件分布。该方法避免了顺序依赖偏差，并生成一个反映所有局部条件妥协的平稳分布。我们在合成分布采样、一致推理任务和贝叶斯结构学习中应用该方法，结果表明，在可通过噪声LLM条件获取的世界先验下，使用LLM条件进行马尔可夫链蒙特卡洛（MCMC）是单次生成的一种可行替代方案。
原文摘要 · Abstract (English)
The knowledge encoded in large language models (LLMs) can serve as a substrate for structured reasoning over variables describing a complex world, but accessing this knowledge in a probabilistically coherent manner poses a difficult inference problem. We propose Large Language Gibbs, a scheme for structured probabilistic inference that uses conditional distributions of an LLM as transition operators. Rather than sampling structured objects through single-pass autoregressive generation, we iteratively resample individual variables conditioned on others using an LLM's next-token conditionals. This approach avoids order-dependent biases and produces a stationary distribution that reflects a compromise between all local conditionals. We apply this approach to sampling from synthetic distributions, consistent reasoning tasks, and Bayesian structure learning. The results suggest that the use of LLM conditionals in MCMC is a practical alternative to one-pass generation for structured probabilistic inference under a world prior accessible through noisy LLM conditionals.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。