通过融合对齐大模型，实现更助人、更无害、更诚实的对话效果。
H3Fusion: Helpful, Harmless, Honest Fusion of Aligned LLMs
- 采用专家混合机制，在表示空间中可控地调整对齐方向。
- 在三个数据集上比单一对齐模型提升11.37%，鲁棒性更强。
- 适合需要高可信度对话系统的开发者与研究者使用。
预训练大语言模型的对齐持续受到产业界与学术界的关注，旨在确保输出内容具有帮助性、无害性和真实性。然而，在模型表示子空间中找到同时满足这三项属性的点仍具挑战。H3Fusion提出一种基于专家混合（MoE）的融合机制，将对齐建模为子空间内的可控漂移，并通过漂移正则化损失平衡不同对齐维度。此外，通过优化生成嵌入与对齐嵌入之间的距离，并引入门控损失以调控贡献专家的激活，实现更优对齐。在三个基准数据集上的广泛评估表明，H3Fusion在三个方面均优于单个对齐模型11.37%，相比当前最先进的模型集成方法提升13.77%，相较于模型融合方法提升6.18%。代码已开源：https://github.com/git-disl/h3fusion。
原文摘要 · Abstract (English)
The alignment of pre-trained LLMs continues to draw significant attention from both industry and academia, aiming to ensure responses that are helpful, harmless, and honest. However, identifying a point in the model's representation subspace that simultaneously satisfies all these properties remains challenging. H3Fusion addresses this challenge by introducing a mixture-of-experts (MoE)-based fusion mechanism that models alignment as a controllable drift within the subspace, guided by a drift-regularization loss to balance competing alignment dimensions. Furthermore, we formulate the alignment by finding a dual objective of harnessing the distance of generated embeddings and alignment embeddings, and introduce a gating loss by canalizing the activations on the contributing experts. Extensive evaluations of three benchmark datasets show that H3Fusion is more helpful, less harmful, and more honest in three aspects: it outperforms each individually aligned model by 11.37%, and provides stronger robustness compared to the state-of-the-art LLM ensemble approaches by 13.77% and model-merging approaches by 6.18%. Code is available at https://github.com/git-disl/h3fusion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。