用信息价值替代不确定性,智能分配计算资源提升高效模型性能
Uncertainty Is Not Enough: Value-of-Information Routing for Mixtures of LoRA Experts
- 基于信息价值而非不确定性动态分配专家计算资源
- 在相同算力下,准确率提升1.8%,尾延迟降低37%
- 适合追求高精度与低延迟的部署场景
混合低秩适配器专家(MoLE)通过路由机制将输入分配给部分适配器,实现参数高效的扩展。现有动态路由策略在预测不确定时激活更多专家,但这种做法将不确定性等同于有价值的信息:实际上,不确定性可能源于专家间一致的模糊性,而非可被补充的信息。本文提出VI-MoLE,将路由建模为可验证的信息价值分配问题。该方法通过估计每个专家前缀的反事实风险,在校准数据上生成同步的风险上界证书,并根据单位成本下的最大认证边际风险下降,全局分配适配器预算。最终通过终端证书决定是否回答或拒绝。相比传统不确定性门控,该方法能区分当前模糊性、可恢复风险与残余风险。理论证明了证书的同步有效性、在递减收益下的最优贪心分配及估值误差下的分配遗憾。评估涵盖匹配算力下的准确率、证书覆盖率、风险-覆盖率、分布偏移和尾延迟,对比固定与动态MoE-LoRA路由。
原文摘要 · Abstract (English)
Mixtures of low-rank adaptation experts increase parameter-efficient capacity by routing each input through a subset of adapters. Recent dynamic routers activate more experts when the router or prediction is uncertain. This rule silently equates uncertainty with useful additional computation: an uncertain example may contain complementary, unqueried expert evidence, but it may instead remain ambiguous after every expert agrees. We formulate routing as certified value-of-information allocation. VI-MoLE learns the counterfactual risk remaining after each expert prefix, converts these predictions into simultaneous upper-risk certificates on held-out calibration data, and spends a global adapter budget on the token--layer action with the largest certified marginal risk reduction per unit cost. A terminal certificate then decides whether to answer or abstain. Unlike an uncertainty gate, this procedure distinguishes present ambiguity from recoverable and residual risk. We prove simultaneous certificate validity, optimal greedy allocation under diminishing certified gains, and allocation regret under value-estimation error. The evaluation protocol tests matched-compute accuracy, certificate coverage, risk--coverage, distribution shift, and tail latency against fixed and dynamic MoE-LoRA routers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。