用进化策略提升大模型解题多样性,让探索更全面。
Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategies
- 用随机扰动优化模型权重,不依赖梯度的进化策略
- 在数学任务上pass@k比强化学习高37%,解法覆盖更广
- 适合需要多角度探索的科研、数学等发现型任务
大语言模型在数学与科学等发现领域应用日益广泛。常规做法是让模型生成一个最优答案作为解法,但通过增加推理时计算量可进一步提升发现能力。在所谓pass@k机制中,模型被允许生成多种候选解以探索解空间。然而,传统强化学习(RL)后训练会缩小模型输出分布,导致解法覆盖率下降。本文采用无梯度、基于种群的进化策略(ES),直接在权重空间进行随机扰动优化。实验表明,ES在多个数学基准上均实现更高pass@k值,输出分布更宽,解法覆盖更广,从而显著提升性能。因此,对于需多样化探索的发现类任务,ES比强化学习更优。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly deployed in discovery domains such as math and science. The usual approach is to present the problem to the model and use its answer as the proposed solution. However, beyond this best guess, discovery can be enhanced by increasing test-time compute. In a process called pass@k, the model is allowed to explore the solution space and generate diverse candidate solutions. Unfortunately, the standard approach to post-training LLMs through Reinforcement Learning (RL) may limit pass@k: the model's output distribution narrows around high-reward outputs, causing the solution coverage to collapse. The alternative is to use Evolution Strategies (ES), a population-based, gradient-free post-training method that optimizes directly in weight space through random perturbations. As this paper shows, ES achieves consistently higher pass@k than RL and produces a broader output distribution with greater solution coverage. This coverage in turn makes it possible to achieve better results in e.g. standard math benchmarks. Thus, ES provides a better foundation for post-training in discovery problems and other domains where diverse solution coverage is critical.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。