将专家模型与潜空间注意力结合，实现高效语言建模。
Unifying Mixture of Experts and Multi-Head Latent Attention for Efficient Language Models
- 用64个微专家和顶k选择实现细粒度路由，支持超百万种组合。
- 保留2个通用专家常驻，仅路由6个专用专家，降低冗余计算。
- 无梯度冲突的负载均衡，推理速度提升3.2倍且内存减少68%。
我们提出MoE-MLA-RoPE，一种融合混合专家（MoE）、多头潜空间注意力（MLA）与旋转位置嵌入（RoPE）的新型架构，以解决模型容量与计算效率之间的根本矛盾。该方法通过三项创新实现高效建模：(1) 采用64个微专家及顶-k选择机制，实现灵活专业化，支持3.6×10⁷种专家组合；(2) 共享专家隔离策略，固定2个专家始终活跃处理通用模式，仅路由6个从62个专家中选出的专用专家；(3) 无梯度冲突的负载均衡机制，在不干扰主损失优化的前提下维持专家利用率。在17M至202M参数的模型上实验表明，当压缩比r=d/2时，该方法实现68%的KV缓存内存减少与3.2倍推理加速，同时保持竞争力困惑度（仅0.8%下降）。相比53.9M参数的原始Transformer，MoE-MLA-RoPE在验证损失上提升6.9%，每前向传播仅使用42%的活跃参数。在FLOP匹配条件下，性能提升达11.1%，推理加速3.2倍。GPT-4自动评估显示生成质量提升，连贯性得分8.1/10，创意性7.9/10，语法正确性8.2/10。结果表明，架构创新而非参数扩展，才是资源受限场景下语言模型效率的前沿。
原文摘要 · Abstract (English)
We present MoE-MLA-RoPE, a novel architecture combination that combines Mixture of Experts (MoE) with Multi-head Latent Attention (MLA) and Rotary Position Embeddings (RoPE) for efficient language modeling. Our approach addresses the fundamental trade-off between model capacity and computational efficiency through three key innovations: (1) fine-grained expert routing with 64 micro-experts and top-$k$ selection, enabling flexible specialization through 3.6 * 10^7 possible expert combinations; (2) shared expert isolation that dedicates 2 always active experts for common patterns while routing to 6 of 62 specialized experts; and (3) gradient-conflict-free load balancing that maintains expert utilization without interfering with primary loss optimization. Extensive experiments on models ranging from 17M to 202M parameters demonstrate that MoE-MLA-RoPE with compression ratio r=d/2 achieves 68% KV cache memory reduction and 3.2x inference speedup while maintaining competitive perplexity (0.8% degradation). Compared to the parameters with 53.9M parameters, MoE-MLA-RoPE improves the validation loss by 6.9% over the vanilla transformers while using 42% fewer active parameters per forward pass. FLOP-matched experiments reveal even larger gains: 11.1% improvement with 3.2x inference acceleration. Automated evaluation using GPT-4 as a judge confirms quality improvements in generation, with higher scores on coherence (8.1/10), creativity (7.9/10) and grammatical correctness (8.2/10). Our results establish that architectural novelty, not parameter scaling, defines the efficiency frontier for resource-constrained language model deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。