提出两种采样方法，让大模型生成更优文本且无需额外奖励模型。
Recipes for Steering and Scaling LLMs via Sampling
- 基于SMC和重置换采样，动态调整生成分布。
- 在不依赖外部监督下，生成质量优于Best-of-N和传统MCMC。
- 适合追求高质量生成但无标注数据的场景。
大型语言模型是概率模型，通常由自回归因子分解定义。尽管近期研究开始探索超越基础模型的更丰富目标分布，但现有采样策略仍效率低下。本文提出一种灵活且理论完备的框架，用于通过采样实现对自回归大模型的引导与扩展。在此框架内，我们介绍了两种算法：一种基于序贯蒙特卡洛（SMC），另一种基于重置换（RE），可引导生成过程趋向于幂、乘积或倾斜的基础分布。通过该框架，我们在无需外部监督或奖励模型的情况下提升了大模型的生成质量。实验表明，我们的方法在扩展性上优于Best-of-N和标准MCMC基线。整体而言，本文为基于采样的大模型概率推断提供了系统化方案。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are probabilistic models, typically defined by an autoregressive factorization. While recent work has begun to study richer target distributions beyond the base model, the sampling strategies remain highly inefficient. In this paper, we present a flexible and theoretically grounded framework for steering and scaling autoregressive LLMs with sampling. Within this framework, we describe two algorithms -- one based on Sequential Monte Carlo (SMC) and one based on Replica Exchange (RE) -- that steer generation toward powering, product or tilting of the base model distribution. We illustrate this framework through scaling the generation quality of LLMs without external supervision or reward models. Experimental results demonstrate our methods scale more favorably than Best-of-N and standard MCMC baselines. Overall, this paper offers a systematic recipe for probabilistic inference with LLMs via sampling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。