arXiv:2605.11153cs.CLcs.LG2026-05被引 1

拆解进化型LoRA架构,发现路由机制是性能关键。

Decomposing Evolutionary Mixture-of-LoRA Architectures: The Routing Lever, the Lifecycle Penalty, and a Substrate-Conditional Boundary

  • 提出路由重写机制,基于隐藏状态动态控制适配器激活
  • 实验证明路由改进带来0.0426 nat对数困惑度提升
  • 仅当适配器预对齐任务时,进化搜索才有效,否则反而拖累性能

我们在一个从零开始的约150M参数扩增型底座(D=1536, V=32000;D/V≈0.048,“widened-1536”底座)上,将进化型LoRA系统分解为三个因素:路由重写(基于隐藏状态的并行Sigmoid门,含可学习每适配器下界与有界温度退火)、领域留一评估范围、以及生命周期(死亡+alpha混合继承+ SVD突变+槽重分配)。报告了在n=3种子和每单元25000次适应步骤下的5/8部分2^3因子设计实验。在该底座上,归因链清晰:路由重写贡献了全部+0.0426 nat的平衡对数困惑度改进(Delta = log PPL_ref - log PPL_test,正值表示提升;t=12.86, p=0.006),归因于“完整进化系统对比静态B3基线”;而系统整体对比基线的提升仅为+0.015 nats(t=1.94, p=0.19),未通过α=0.05显著性检验。领域评估范围在种子级别无显著影响,生命周期则为净负效应约-0.028 nats(t=-4.46, p=0.047)。辅助的α=0继承反事实在主线指标上方向不一致,且统计功效不足,无法得出等效或支撑性结论(此前使用算术均值聚合器错误通过;见附录B.11)。基线扰动探针从方向上否定了生命周期的“基因组上下文”重构解释。可控合成沙盒定位出底座条件下的临界边界:仅当适配器预对齐任务时,路由通道的进化搜索才起关键作用;在其他所有测试情形中,其表现均劣于、持平或主动损害梯度解。

原文摘要 · Abstract (English)

We decompose an evolutionary mixture-of-LoRA system on a from-scratch ~150M-parameter widened-D substrate (D=1536, V=32000; D/V approx 0.048; the "widened-1536" substrate) into three factors -- a router rewrite (parallel sigmoid gate with learnable per-adapter floor and bounded temperature anneal, fed post-stack hidden states rather than token-embedding means), a per-domain leave-one-out evaluation scope, and a lifecycle of death plus alpha-blend inheritance plus SVD mutation plus slot reallocation -- and report a 5-of-8 partial 2^3 factorial run at n=3 seeds and 25000 adaptation steps per cell. The attribution chain is sharp on this substrate: the router rewrite carries the entire +0.0426 nat balanced log-PPL improvement (Delta = log PPL_ref - log PPL_test, positive = improvement; t=12.86, p=0.006) attributed to "the full evolutionary system vs the static B3 baseline"; the headline full-system-vs-B3 balanced contrast itself is +0.015 nats, t=1.94, p=0.19 at n=3 and does not clear alpha=0.05. The per-domain evaluation scope is null at seed-resolution, and the lifecycle is a net drag of approx -0.028 nats (t=-4.46,p=0.047 in the primary chain). An auxiliary alpha=0 inheritance counterfactual at n=3 seeds is sign-inconsistent at the headline metric and underpowered for either an equivalence or load-bearing conclusion (corrected from an earlier arithmetic-mean aggregator that erroneously cleared inheritance; see Appendix B.11). A base-perturbation probe directionally refutes a "genomic-context" reframe of the lifecycle role. A controllable synthetic sandbox locates a substrate-conditional regime boundary: evolutionary search on the routing channel is load-bearing only when adapters are pre-aligned to the task; in every other regime tested it underperforms, ties, or actively degrades the gradient solution.

LoRA进化计算模型优化路由机制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。