不加参数就能提升模型融合效果，靠的是修正特征空间的旋转偏差。
MOMA: Masked Orthogonal Matrix Alignment for Zero-Additional-Parameter Model Merging
- 用全局掩码和任务专属正交变换修复特征对齐问题。
- 在分类任务上性能接近顶尖方法，且无需新增参数或计算开销。
- 适合想低成本融合多个模型又不想增加推理负担的研究者。
模型融合为多任务学习提供了一种可扩展的替代方案，但常导致分类任务性能下降。我们发现其根源在于融合后的编码器与固定的任务特定分类头之间存在几何错位。现有方法通常依赖辅助参数强制严格表征对齐，但我们揭示这种错位主要是正交变换，因此严格对齐并非必要。基于此，我们提出MOMA（Masked Orthogonal Matrix Alignment），通过联合优化全局多任务向量掩码和任务特定的正交变换来修正错位。关键在于，MOMA将对应的新参数直接融入原有模型权重中，实现了与最先进基线相当的性能，同时零新增参数、零额外推理成本。
原文摘要 · Abstract (English)
Model merging offers a scalable alternative to multi-task learning but often yields suboptimal performance on classification tasks. We attribute this degradation to a geometric misalignment between the merged encoder and static task-specific classifier heads. Existing methods typically rely on auxiliary parameters to enforce strict representation alignment. We challenge this approach by revealing that the misalignment is predominantly an orthogonal transformation, rendering such strict alignment unnecessary. Leveraging this insight, we propose MOMA (Masked Orthogonal Matrix Alignment), which rectifies the misalignment by jointly optimizing a global multi-task vector mask and task-specific orthogonal transformations. Crucially, MOMA absorbs corresponding new parameters directly into the existing model weights, achieving performance comparable to state-of-the-art baselines with zero additional parameters and zero added inference cost.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。