解决大模型对齐中的多目标冲突问题，提升有用、无害、诚实表现。
When the Model Said 'No Comment', We Knew Helpfulness Was Dead, Honesty Was Alive, and Safety Was Terrified
- 分两阶段优化：先提取任务特征，再用几何校准专家路由。
- 在多个数据集上显著提升有用性、诚实性和安全性，错误率下降4.3%。
- 适合关注模型对齐与安全性的研究人员及工程师。
大型语言模型需符合人类价值观——有用、无害、诚实（HHH），这对安全部署至关重要。现有方法使用监督微调（SFT）和混合专家（MoE）进行对齐，但在多目标场景下面临挑战：SFT导致目标间干扰，MoE存在路由不准问题。我们称此为轴坍缩，表现为特征空间割裂引发灾难性遗忘，以及错误路由导致推理不可靠。为此提出AlignX框架：第一阶段通过提示注入微调提取特定任务特征，缓解遗忘；第二阶段采用MoCaE模块，利用分形与自然几何校准专家路由，提升推理可靠性。AlignX在Alpaca（有用性）、BeaverTails（无害性）和TruthfulQA（诚实性）上取得显著提升，分别实现+171.5%胜率、+110.1%真实性-信息量得分，安全违规减少4.3%。相比先前MoE，延迟与内存占用降低超35%。四种不同大模型验证了其通用性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) need to be in accordance with human values-being helpful, harmless, and honest (HHH)-is important for safe deployment. Existing works use Supervised Fine-Tuning (SFT) and Mixture-of-Experts (MoE) to align LLMs. However, these works face challenges in multi-objective settings, such as SFT leading to interference between conflicting objectives, while MoEs suffer from miscalibrated routing. We term this failure mode Axis Collapse, marked by (1) disjoint feature spaces causing catastrophic forgetting, and (2) unreliable inference from misrouted experts. To resolve this, we propose AlignX, a two-stage framework. Stage 1 uses prompt-injected fine-tuning to extract axis-specific task features, mitigating catastrophic forgetting. Stage 2 deploys a MoCaE module that calibrates expert routing using fractal and natural geometry, improving inference reliability. AlignX achieves significant gains on Alpaca (Helpfulness), BeaverTails (Harmlessness), and TruthfulQA (Honesty), with +171.5% win rate, +110.1% in truthfulness-informativeness, and 4.3% fewer safety violations. It also reduces latency and memory usage by over 35% compared to prior MoEs. Results across four LLMs validate its generalizability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。