用字节片段熵指导专家路由，让大模型按需计算。
EntropyMoE: Entropy-Aware Sparse Expert Routing for Tokenizer-Free LLMs
- 根据字节片段熵动态选择专家，实现稀疏计算。
- 在相同下游任务上，比特/字节更低，性能相当。
- 适合追求高效推理的无分词器大模型研究者。
近期的字节级大语言模型通过将字节动态分块，使无分词器建模更具竞争力。然而，现有字节块架构仍对每个块施加相同的密集前馈计算，无法适应块语义与粒度的变化。为此，我们提出EntropyMoE，一种专为动态字节块设计的专家混合（MoE）架构。该架构将全局块Transformer中的密集前馈模块替换为Top-K专家层，以动态块作为专家路由的基本单元，其字节覆盖范围决定工作量分配。路由器直接利用块熵进行专家选择，使用与动态块构建相同的粒度信号来组织稀疏计算。块熵与长度共同定义专家特化的特征空间。实验表明，EntropyMoE在匹配的密集与稀疏基线中实现了最低的保留比特/字节，同时保持相近的下游准确率。结果证实块熵是稀疏条件计算的有效路由坐标，并将专家混合建模扩展至无分词器表示。
原文摘要 · Abstract (English)
Recent byte-level large language models (LLMs) have made tokenizer-free modeling increasingly competitive by grouping bytes into dynamically sized patches. However, existing byte-patch architectures still apply the same dense feed-forward computation to every patch. This uniform computation cannot adapt model capacity to variations in patch semantics and granularity. We address this limitation with EntropyMoE, a Mixture-of-Experts (MoE) architecture designed for dynamic byte patches. EntropyMoE replaces the dense feed-forward modules in the global patch Transformer with Top-K expert layers. Each dynamic patch serves as the basic unit of expert routing, and its byte coverage determines its contribution to workload accounting. The router selects experts directly from patch entropy, using the same granularity signal that underlies dynamic patch construction to organize sparse computation. Patch entropy and length jointly define the feature space for regulating expert specialization. Experiments show that EntropyMoE achieves the lowest held-out bits-per-byte among matched dense and sparse baselines while maintaining comparable downstream accuracy. These results establish patch entropy as an effective routing coordinate for sparse conditional computation and extend Mixture-of-Experts modeling beyond tokenizer-based representations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。