用进化算法让大模型对齐更多样，避免单一行为模式。
EvoPref: Multi-Objective Evolutionary Optimization Discovers Diverse LLM Alignments Beyond Gradient Descent
- 采用多目标进化算法优化多个对齐目标，保持多样性。
- 相比梯度方法，偏好覆盖提升18%，崩溃率降低47%。
- 适合需要多样化输出的场景，如内容生成与安全对齐。
基于梯度的大型语言模型（LLM）对齐方法易出现偏好坍缩，收敛于狭窄行为模式而忽略多样性。本文提出EvoPref，一种基于非支配排序遗传算法II（NSGA-II）的多目标进化算法，通过带档案库的多样性保持机制，对低秩适配器（LoRA）在帮助性、无害性和诚实性三个目标上进行优化。实验表明，群体方法显著提升了对齐多样性：在标准基准上，偏好覆盖率提升18%（中位数82.5%对比ORPO的70.0%，p<0.001，Wilcoxon检验，n=30），坍缩率下降47%（11.0%对比20.6%，p<0.001）。同时保持良好对齐质量（中位数75.5% RewardBench对比ORPO的75.0%，p<0.05）。理论分析结合近期多目标进化算法（MOEA）运行时间研究（Dang et al., 2025），解释了档案库方法为何更有效避免坍缩。与MOEA/D、SMS-EMOA、CMA-ES及梯度基线（DPO、IPO、KTO、ORPO）的全面比较，经严格统计检验（Friedman检验+Holm校正、Vargha-Delaney效应量、中位数与四分位距）确认，多目标选择结合多样性保持至关重要。本工作确立进化优化为实现多样化大模型对齐的系统化范式。
原文摘要 · Abstract (English)
Gradient-based preference optimization methods for large language model (LLM) alignment suffer from preference collapse, converging to narrow behavioral modes while neglecting preference diversity. We introduce EvoPref, a multi-objective evolutionary algorithm that maintains populations of Low-Rank Adaptation (LoRA) adapters optimized across helpfulness, harmlessness, and honesty objectives using Non-dominated Sorting Genetic Algorithm II (NSGA-II) selection with archive-based diversity preservation. Our primary contribution is demonstrating that population-based methods discover substantially more diverse alignments than gradient descent. On standard benchmarks, EvoPref improves preference coverage by 18% (median 82.5% vs. 70.0% for ORPO, $p<0.001$, Wilcoxon, $n=30$) and reduces collapse rates by 47% (11.0% vs. 20.6%, $p<0.001$), while achieving competitive alignment quality (median 75.5% RewardBench vs. 75.0% for ORPO, $p<0.05$). We provide theoretical motivation extending recent multi-objective evolutionary algorithm (MOEA) runtime analysis (Dang et al., 2025) suggesting why archive-based methods escape collapse more effectively than single-trajectory optimization. Comprehensive comparisons against MOEA/D, SMS-EMOA, CMA-ES, and gradient baselines (DPO, IPO, KTO, ORPO) with rigorous statistical testing (Friedman with Holm correction, Vargha-Delaney effect sizes, median with IQR) confirm that multi-objective selection with diversity preservation is essential. This work establishes evolutionary optimization as a principled paradigm for diverse LLM alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。