通过自蒸馏优化语言生成，提升受限文本质量
Improving Constrained Language Generation via Self-Distilled Twisted Sequential Monte Carlo
- 用自蒸馏迭代优化基础模型，增强与目标分布的对齐
- 在稀疏奖励下生成质量显著提升，解决难样本生成问题
- 适合需要高质量受限生成的应用场景，如安全内容生成
近期工作将自回归语言模型的受限文本生成视为概率推断问题。其中，Zhao 等（2024）提出基于扭曲序列蒙特卡洛的方法，引入学习到的扭曲函数和扭曲诱导的提案来引导生成过程。然而，在目标分布集中在基础模型难以生成的输出时，由于奖励信号稀疏且无信息量，学习变得困难。本文表明，通过自蒸馏迭代优化基础模型，可逐步提升其与目标分布的对齐程度，从而显著提升生成质量。
原文摘要 · Abstract (English)
Recent work has framed constrained text generation with autoregressive language models as a probabilistic inference problem. Among these, Zhao et al. (2024) introduced a promising approach based on twisted Sequential Monte Carlo, which incorporates learned twist functions and twist-induced proposals to guide the generation process. However, in constrained generation settings where the target distribution concentrates on outputs that are unlikely under the base model, learning becomes challenging due to sparse and uninformative reward signals. We show that iteratively refining the base model through self-distillation alleviates this issue by making the model progressively more aligned with the target, leading to substantial gains in generation quality.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。