提出AlignMerge框架，让大模型合并时保持安全对齐性。
AlignMerge - Alignment-Preserving Large Language Model Merging via Fisher-Guided Geometric Constraints
- 基于费舍尔几何约束，显式保护对齐方向
- 在5个模型族上提升对齐质量且不牺牲能力
- 适合需要安全可控模型合并的开发者
大模型合并是无需重训练即可融合多个微调版本能力的有效方法。然而，现有方法（如线性权重汤、任务向量、费舍尔加权平均）虽能维持损失，却可能悄然破坏对齐性。本文认为合并应是在已对齐锚点附近的几何约束操作，而非事后验证。为此提出AlignMerge框架，将对齐作为显式不变量：在指令微调基模型的局部费舍尔坐标系中，通过投影矩阵P_A估计对齐子空间，并优化目标函数：L_AlignMerge = L_geo + λ_align * L_align + λ_bud * L_bud，其中L_geo保持合并结果靠近专家在费舍尔-劳几何中的位置，L_align惩罚沿对齐敏感方向的偏移，L_bud施加软对齐预算。使用解码无关的对齐质量指数（AQI）作为对齐度量，该指标捕捉表示空间中对齐与非对齐行为的分离清晰度。在五个模型家族（LLaMA-3 8B、Mistral 7B、Qwen 2、Phi-3.5、Gemma 2）上，将安全锚点与任务专家合并，AlignMerge在对齐指标（AQI、毒性、LLM判别对齐）上优于或等同于最佳专家，同时在指令遵循、推理和助人能力上表现相当或更优。其对齐子空间漂移更小，预算违规更少，优于费舍尔汤、TIES、SafeMerge和MergeAlign。结果表明，对齐保持型合并应成为设计首要目标，为未来基础模型的几何感知组合提供路径。
原文摘要 · Abstract (English)
Merging large language models (LLMs) is a practical way to compose capabilities from multiple fine-tuned checkpoints without retraining. Yet standard schemes (linear weight soups, task vectors, and Fisher-weighted averaging) can preserve loss while quietly destroying alignment. We argue that merging is not a numerical trick but a geometry-constrained operation around an already-aligned anchor: fusion must be steered to respect safety geometry, not validated post hoc. We introduce AlignMerge, a geometry-aware merging framework that makes alignment an explicit invariant. In a local Fisher chart around an instruction-tuned base, we estimate an alignment subspace with projector P_A and optimize: L_AlignMerge = L_geo + lambda_align * L_align + lambda_bud * L_bud, where L_geo keeps the merge close to its experts in Fisher-Rao geometry, L_align penalizes motion along alignment-sensitive directions, and L_bud enforces a soft alignment budget. As the alignment functional we use the decoding-invariant Alignment Quality Index (AQI), a latent-space criterion that captures how cleanly aligned and misaligned behaviors separate in representation space. Across five model families (LLaMA-3 8B, Mistral 7B, Qwen 2, Phi-3.5, Gemma 2), merging safety anchors with task experts, AlignMerge improves alignment metrics (AQI, toxicity, LLM-judge alignment) while matching or exceeding the best expert on instruction-following, reasoning, and helpfulness. It also exhibits smaller alignment-subspace drift and fewer budget violations than Fisher soups, TIES, SafeMerge, and MergeAlign. These results make alignment-preserving merging a first-class design goal and suggest a path to geometry-aware composition of future foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。