动态选择专家序列，让计算随推理逐步加深，提升精度。
Mixture of Raytraced Experts
- 通过逐轮采样专家构建可变深度计算图
- 训练周期减少10%至40%，精度相当或更高
- 无需负载均衡，适合追求高效推理的场景
我们提出一种基于光线追踪的专家混合模型（Mixture of Raytraced Experts），是一种堆叠式混合专家（MoE）架构，能够动态选择专家序列，生成宽度和深度可变的计算图。现有MoE架构对每个样本需固定计算量，而本方法使预测精度随专家序列迭代逐步提升。通过从候选专家集中迭代采样并展开序列，类比循环神经网络训练方式完成模型训练。该方法无需负载均衡机制，初步实验表明训练周期减少10%至40%，同时保持或超过原有精度。结果揭示了MoE研究的新方向，有望设计出更快速、更具表现力的模型。代码已开源：https://github.com/nutig/RayTracing
原文摘要 · Abstract (English)
We introduce a Mixture of Raytraced Experts, a stacked Mixture of Experts (MoE) architecture which can dynamically select sequences of experts, producing computational graphs of variable width and depth. Existing MoE architectures generally require a fixed amount of computation for a given sample. Our approach, in contrast, yields predictions with increasing accuracy as the computation cycles through the experts' sequence. We train our model by iteratively sampling from a set of candidate experts, unfolding the sequence akin to how Recurrent Neural Networks are trained. Our method does not require load-balancing mechanisms, and preliminary experiments show a reduction in training epochs of 10\% to 40\% with a comparable/higher accuracy. These results point to new research directions in the field of MoEs, allowing the design of potentially faster and more expressive models. The code is available at https://github.com/nutig/RayTracing
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。