小种群进化策略在大模型微调中表现不佳，根源是奖励设计而非规模限制。
Reward-Aware Population Scaling of Evolutionary Strategies in LLM Fine-Tuning
- 用归一化奖励时小种群易失效，去归一化后N=2也能提升性能。
- 二值奖励下，零优势概率与基础准确率、批量大小相关，可精确计算。
- 适合关注高效微调和奖励机制优化的研究者，尤其在资源受限场景。
使用进化策略（ES）微调大语言模型具有内存效率高、可并行、兼容黑盒或离散奖励等优点。然而，关于种群规模的结论存在显著矛盾：以交叉熵（CE）为奖励时，N=1 即可成功；而采用二值奖励训练时，通常需要 N≈30。本文表明，这一差距主要源于奖励设计与归一化方式，而非种群规模本身。在研究的高能力模型场景中，z-score优势归一化会导致 N=2 失败。去除归一化后，二值奖励下的 ES 在 N=2 时仍能提升 GSM8K 与 TREC 性能，覆盖 0.5B-7B 参数量模型，而归一化版本则崩溃或退化。小种群失败风险由奖励粒度决定：二值准确率奖励的零优势概率 $q$ 可由基础准确率、批大小及成对正确性相关性闭式表达。对 Qwen2.5-Instruct/GSM8K 的零训练探针实验，12 种配置下均方绝对误差仅为 0.020，验证了公式有效性，并发现可用阈值 $N_{\mathrm{avail}}$ 在该场景下较小。这意味着，小种群在高能力模型的二值奖励 ES 中的失败，可能是实现问题而非固有局限。
原文摘要 · Abstract (English)
Using Evolutionary Strategies (ES) for fine-tuning large language models is attractive because it is memory-efficient, parallel, and compatible with black-box or discrete rewards. Yet its population-size conclusions conflict sharply: fine-tuning with cross-entropy (CE) reward succeeds with $N=1$, while binary-reward training often needs $N \approx 30$. We show this gap is largely about reward design and normalization, not population size. In the capable-model regime we study, z-score advantage normalization can cause $N=2$ to fail. Disabling normalization lets binary-reward ES with $N=2$ improve on GSM8K and TREC across capable models spanning 0.5B-7B, where the normalized variant collapses or degrades. This small-$N$ risk is set by reward granularity: binary accuracy reward induces a zero-advantage probability $q$ that depends in closed form on base accuracy, batch size, and intra-pair correctness correlation; a zero-training probe on Qwen2.5-Instruct/GSM8K matches the formula with mean absolute error 0.020 across 12 configurations and finds the availability threshold $N_{\mathrm{avail}}$ to be small in this capable-model regime. The implication is not that $N=2$ is universally sufficient, but that small-population failure in capable-model binary ES can be an implementation artifact rather than an intrinsic population limit.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。