通过光线追踪机制动态激活专家网络，实现更高效低参推理。
Ray-Tracing for Conditionally Activated Neural Networks
- 分层专家模块+渐进采样，动态构建网络路径。
- 推理参数量显著减少，且与输入复杂度正相关。
- 无需额外惩罚项，自动适应输入复杂度，适合轻量化部署。
本文提出一种新型条件激活神经网络架构，结合多层混合专家（MoEs）的分层设计与渐进采样机制，使专家激活配置逐步收敛至优化状态。该方法支持网络结构的动态展开，实现路径特定的高效训练。实验表明，该方法在保持与传统基线相当精度的同时，大幅降低推理所需参数量。值得注意的是，参数减少程度与输入模式复杂度自然相关，这一特性源于网络运行动力学，无需显式引入辅助惩罚函数。
原文摘要 · Abstract (English)
In this paper, we introduce a novel architecture for conditionally activated neural networks combining a hierarchical construction of multiple Mixture of Experts (MoEs) layers with a sampling mechanism that progressively converges to an optimized configuration of expert activation. This methodology enables the dynamic unfolding of the network's architecture, facilitating efficient path-specific training. Experimental results demonstrate that this approach achieves competitive accuracy compared to conventional baselines while significantly reducing the parameter count required for inference. Notably, this parameter reduction correlates with the complexity of the input patterns, a property naturally emerging from the network's operational dynamics without necessitating explicit auxiliary penalty functions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。