用动态路由让不同专家处理长短文本，实现高效精准医疗问答。
MambaFormer: Token-Level Guided Routing Mixture-of-Experts for Accurate and Efficient Clinical Assistance
- 根据输入复杂度动态分配短复杂查询或长序列到专用Transformer或状态空间模型。
- 在DentalQA和PubMedQA上达到0.918的BERTScore，推理延迟仅0.077秒。
- 适合资源受限的临床场景，尤其对实时性要求高的智能问诊系统有参考价值。
大型语言模型在真实医疗应用中受制于计算成本与线性时间模型之间的根本权衡。为此，我们提出一种基于LLM的MambaFormer混合专家（MoE）框架，用于高效医疗问答与临床辅助。该框架采用轻量级门控机制，实现基于令牌的动态路由：短而复杂的查询由定制Transformer专家（ET5）处理，长而高吞吐的序列则交由状态空间模型专家（EMamba）处理。定制化的EMamba与ET5模型针对输入序列维度、嵌入结构、序列长度及目标输出头进行优化，并在新设计的DentalQA数据集上通过迁移学习微调。路由决策由令牌嵌入的上下文复杂度、归一化序列长度及领域感知特征驱动，从而实现推理延迟与预测准确率间的帕累托最优。此外，新型效用引导多目标损失自适应调节令牌级专家激活，联合优化路由决策、路由器参数、路由行为、专家利用率与计算成本。最后，所提MambaFormer在新设计的DentalQA与PubMedQA数据集上进行交叉验证（保留验证），并与最先进方法对比。结果表明，该模型在保持0.9180的BERTScore的同时，实现0.077秒的超低延迟，较T5-Large快24.4倍，为资源受限的临床部署提供可扩展解决方案。
原文摘要 · Abstract (English)
The deployment of large language models (LLMs) in real-world clinical applications is constrained by the fundamental trade-off between computational cost and the efficiency of linear-time models. To address this, we propose an LLM-based MambaFormer hybrid Mixture-of-Experts (MoE) framework for efficient medical question-answering (QA) and clinical assistance. The MambaFormer employs a lightweight gating mechanism that performs token-level dynamic routing to a customized Transformer expert (ET5) for short, complex queries or to a State Space Model expert (EMamba) for long, high-throughput sequences. The customized EMamba and ET5 models are tailored to accommodate input sequence dimensionality, embedding structure, sequence length, and target-specific output heads, and are fine-tuned through transfer learning on a new, custom-designed DentalQA dataset. Moreover, intelligent routing decisions are driven by the contextual complexity of token embeddings, normalized sequence length, and domain-aware features, thereby enforcing a Pareto-optimal trade-off between inference latency and prediction accuracy. Furthermore, a novel utility-guided multi-objective loss jointly optimizes decisions, router parameters, routing behavior, expert utilization, and computational cost by adaptively regulating token-level expert activation. Finally, the proposed MambaFormer is cross-validated (holdout) for medical QA on the new, custom-designed DentalQA and PubMedQA datasets and compared with state-of-the-art techniques. The proposed MambaFormer outperforms (BERTScore = 0.9180) with ultra-low latency (0.077 s), delivering a 24.4 speedup over T5-Large and establishing a scalable solution for resource-constrained clinical deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。