让Transformer神经元像大脑一样灵活切换激活函数，提升模型效率与性能。
PolyGLU: State-Conditional Activation Routing in Transformer Feed-Forward Networks
- 每个前馈层神经元可动态选择4种激活函数，通过输入条件和学习偏好决定使用哪种。
- 深度网络中早期层倾向用GELU，深层强烈偏好Tanh，且路由几乎确定性（熵仅0.03%）。
- 仅增加0.23%参数开销，训练成本极低，适合资源受限场景的高效模型部署。
生物神经系统的突触传递依赖多种神经递质（如谷氨酸、多巴胺）实现不同信号处理功能，而现代Transformer在所有前馈神经元中统一使用单一固定激活函数。本文提出PolyGLU（多色门控线性单元），作为SwiGLU的即插即用替代方案，使每个前馈网络神经元能通过可微分机制在K=4种激活函数间动态路由，该机制结合了学习得到的静态偏好与输入相关的门控信号，采用端到端训练的Gumbel-Softmax方法。我们在约100亿个标记上使用单张NVIDIA A100 GPU训练了一个5.97亿参数的PolychromaticLM模型。关键发现是涌现式路由行为：无需显式的稀疏损失或熵正则化，路由机制自然收敛至近确定性选择（平均动态熵仅为最大值的0.030%），呈现显著的深度依赖特化模式——浅层偏好GELU，深层强烈倾向Tanh。三个层次保持较高路由熵，提示存在计算灵活性节点。该架构仅引入0.23%参数开销（约140万参数），且在长达13,067步的监督微调中始终保持路由熵恒定为ln(4)，表现出完全鲁棒性。在标准基准测试中，尽管仅使用3600倍更少的训练数据，PolychromaticLM仍达到Qwen3-0.6B-Base性能的62%-89%。所有代码、权重及训练基础设施均以Apache 2.0协议开源。
原文摘要 · Abstract (English)
Biological neural systems employ diverse neurotransmitters -- glutamate, GABA, dopamine, acetylcholine -- to implement distinct signal-processing modalities within shared neural circuits. In contrast, modern transformers apply a single fixed activation function across all feed-forward neurons. We introduce PolyGLU (Polychromatic Gated Linear Unit), a drop-in replacement for SwiGLU that enables each FFN neuron to dynamically route among K=4 activation functions via a differentiable mechanism combining learned static preferences with input-conditioned gating, trained end-to-end with Gumbel-Softmax. We train PolychromaticLM, a 597M-parameter transformer, on ~10B tokens using a single NVIDIA A100 GPU. Our key finding is emergent routing behavior: without any explicit sparsity loss or entropy regularization, the routing mechanism converges to near-deterministic activation selections (mean dynamic entropy = 0.030% of maximum), with a striking depth-dependent specialization pattern -- early layers prefer GELU while deep layers strongly favor Tanh. Three layers maintain elevated routing entropy, suggesting computational flexibility points. The routing architecture adds only 0.23% parameter overhead (~1.4M parameters) and proves fully robust to supervised fine-tuning: routing entropy remains constant at ln(4) throughout 13,067 SFT steps. On standard benchmarks, PolychromaticLM achieves 62-89% of Qwen3-0.6B-Base performance despite training on 3,600x fewer tokens. All code, weights, and training infrastructure are released under Apache 2.0.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。