让大模型生成内容严格满足要求,同时保持原有风格。
Guaranteed Generation from Large Language Models
- 用自回归生成+拒绝采样组合方法,保证输出合规。
- 实验显示约束完全满足,且生成风格几乎不变,速度更快。
- 适合对输出可靠性要求高的场景,如安全审查、内容生成。
随着大语言模型在各类应用中日益普及,如何控制文本生成以满足特定约束成为关键问题。本文定义了理想分布——最接近原始模型且始终满足约束的分布——作为生成保障的终极目标。研究发现,仅通过自回归训练无法达到该目标,因此需结合训练与推理阶段的方法。为此提出GUARD框架:将自回归提议分布与拒绝采样结合,理论证明其能同时优化推理速度与分布贴近度。在词汇约束和情感反转两种高难度约束场景下的实验表明,GUARD实现100%约束满足,几乎完整保留理想分布,并显著提升推理效率。该方法为大模型生成提供可靠保障,不牺牲生成能力。
原文摘要 · Abstract (English)
As large language models (LLMs) are increasingly used across various applications, there is a growing need to control text generation to satisfy specific constraints or requirements. This raises a crucial question: Is it possible to guarantee strict constraint satisfaction in generated outputs while preserving the distribution of the original model as much as possible? We first define the ideal distribution - the one closest to the original model, which also always satisfies the expressed constraint - as the ultimate goal of guaranteed generation. We then state a fundamental limitation, namely that it is impossible to reach that goal through autoregressive training alone. This motivates the necessity of combining training-time and inference-time methods to enforce such guarantees. Based on this insight, we propose GUARD, a simple yet effective approach that combines an autoregressive proposal distribution with rejection sampling. Through GUARD's theoretical properties, we show how controlling the KL divergence between a specific proposal and the target ideal distribution simultaneously optimizes inference speed and distributional closeness. To validate these theoretical concepts, we conduct extensive experiments on two text generation settings with hard-to-satisfy constraints: a lexical constraint scenario and a sentiment reversal scenario. These experiments show that GUARD achieves perfect constraint satisfaction while almost preserving the ideal distribution with highly improved inference efficiency. GUARD provides a principled approach to enforcing strict guarantees for LLMs without compromising their generative capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。