通过迭代优化价值函数，提升大模型生成质量与效率。
Evolutionary Guided Decoding: Iterative Value Refinement for LLMs
- 采用进化式框架，逐步优化价值函数的训练数据分布。
- 在摘要、对话等任务中显著提升对齐效果，降低计算成本。
- 适合需要高效可控生成的场景，如对话系统与内容创作。
尽管引导解码（尤其是基于价值的方法）已成为无需微调即可控制语言模型输出的低成本方案，但其效果受限于价值函数的准确性。我们发现，这种不准确源于核心的分布差距：现有方法仅在基础策略采样的轨迹上训练静态价值函数，导致其训练局限于狭窄且次优的输出空间。为此，我们提出迭代价值精炼（Iterative Value Refinement），一个旨在缩小该差距的进化框架。它通过价值探索获取更全面、鲁棒的训练信号，并结合迭代自精炼机制，利用前一轮改进后的价值函数指导下一轮高质量数据生成。在文本摘要、多轮对话和指令遵循任务上的大量实验表明，该框架能有效实现模型对齐。本方法不仅达成高质量对齐，还通过合理的价值函数优化显著降低计算开销，实现高效精准的控制。
原文摘要 · Abstract (English)
While guided decoding, especially value-guided methods, has emerged as a cost-effective alternative for controlling language model outputs without re-training models, its effectiveness is limited by the accuracy of the value function. We identify that this inaccuracy stems from a core distributional gap: existing methods train static value functions on trajectories sampled exclusively from the base policy, which inherently confines their training to a narrow and suboptimal view of the potential output space. We propose Iterative Value Refinement, a evolutionary framework designed to narrow this gap. It employs Value Exploration to provide a more comprehensive and robust training signal, complemented by Iterative Self-Refinement, which uses the improved value function from one iteration to guide the generation of higher-quality data for the next. Extensive experiments on text summarization, multi-turn dialogue, and instruction following demonstrate the effectiveness of our framework in aligning language models. Our approach not only achieves alignment but also significantly reduces computational costs by leveraging principled value function optimization for efficient and effective control.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。