用可校准的专家路由,让大模型更懂帮助、安全和诚实。
Too Helpful, Too Harmless, Too Honest or Just Right?
- 分专家分别处理帮助性、安全性、诚实性,通过校准路由融合输出。
- 在三个基准上分别提升32.5%、33.9%、28.4%的得分表现。
- 比传统MoE少40%内存和延迟,适配多种大模型架构。
大型语言模型在多项NLP任务中表现优异,但使其输出符合帮助性、无害性和诚实性(HHH)仍具挑战。现有方法常孤立优化单一维度,导致权衡与行为不一致。虽然混合专家(MoE)架构具备模块化优势,但路由机制校准不足,限制了其对齐效果。我们提出TrinityX,一种基于校准专家混合(MoCaE)的模块化对齐框架,将各HHH维度的独立训练专家嵌入Transformer结构,通过任务自适应的校准路由整合专家信号,生成统一的对齐感知表示。在三个标准对齐基准上——Alpaca(帮助性)、BeaverTails(无害性)、TruthfulQA(诚实性)——实验证明,TrinityX显著优于强基线,相对胜率提升32.5%,安全评分提高33.9%,真实性提升28.4%。此外,相比以往MoE方法,其内存占用和推理延迟降低超40%。消融实验表明校准路由至关重要,跨模型评估也证实其在多种大模型主干上的泛化能力。
原文摘要 · Abstract (English)
Large Language Models (LLMs) exhibit strong performance across a wide range of NLP tasks, yet aligning their outputs with the principles of Helpfulness, Harmlessness, and Honesty (HHH) remains a persistent challenge. Existing methods often optimize for individual alignment dimensions in isolation, leading to trade-offs and inconsistent behavior. While Mixture-of-Experts (MoE) architectures offer modularity, they suffer from poorly calibrated routing, limiting their effectiveness in alignment tasks. We propose TrinityX, a modular alignment framework that incorporates a Mixture of Calibrated Experts (MoCaE) within the Transformer architecture. TrinityX leverages separately trained experts for each HHH dimension, integrating their outputs through a calibrated, task-adaptive routing mechanism that combines expert signals into a unified, alignment-aware representation. Extensive experiments on three standard alignment benchmarks-Alpaca (Helpfulness), BeaverTails (Harmlessness), and TruthfulQA (Honesty)-demonstrate that TrinityX outperforms strong baselines, achieving relative improvements of 32.5% in win rate, 33.9% in safety score, and 28.4% in truthfulness. In addition, TrinityX reduces memory usage and inference latency by over 40% compared to prior MoE-based approaches. Ablation studies highlight the importance of calibrated routing, and cross-model evaluations confirm TrinityX's generalization across diverse LLM backbones.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。