不同结构的混合模型，该把LoRA放在哪里，效果差别巨大。
Where Should LoRA Go? Component-Type Placement in Hybrid Language Models
- 按组件类型区分放置LoRA，而非统一使用。
- 仅微调注意力模块就能达全模型效果，参数少5-10倍。
- 并行结构可正向迁移，串行结构会灾难性遗忘。
混合语言模型通过交替使用注意力与循环组件，日益具备与纯Transformer相媲美的性能，但现有LoRA方法对所有组件一视同仁，未考虑其功能差异。本文系统研究了两种混合架构（Qwen3.5-0.8B：序列式，GatedDeltaNet + softmax注意力；Falcon-H1-0.5B：并行式，Mamba-2 SSM + 注意力）中组件类型对LoRA位置的影响，在三个领域微调并评估五个基准。结果表明，尽管注意力路径占少数，但仅微调它即可在多数任务上超越全模型微调，且参数量减少5-10倍。关键发现：在序列型混合模型中，微调循环主干会导致性能下降14.8个百分点（如GSM8K），而在并行型中则提升8.6个百分点。此外，发现并行模型具有正向跨任务迁移能力，而序列模型则出现灾难性遗忘。这些结果表明，混合拓扑结构从根本上决定微调响应特性，组件感知的LoRA放置是混合架构设计的必要维度。
原文摘要 · Abstract (English)
Hybrid language models that interleave attention with recurrent components are increasingly competitive with pure Transformers, yet standard LoRA practice applies adapters uniformly without considering the distinct functional roles of each component type. We systematically study component-type LoRA placement across two hybrid architectures -- Qwen3.5-0.8B (sequential, GatedDeltaNet + softmax attention) and Falcon-H1-0.5B (parallel, Mamba-2 SSM + attention) -- fine-tuned on three domains and evaluated on five benchmarks. We find that the attention pathway -- despite being the minority component -- consistently outperforms full-model adaptation with 5-10x fewer trainable parameters. Crucially, adapting the recurrent backbone is destructive in sequential hybrids (-14.8 pp on GSM8K) but constructive in parallel ones (+8.6 pp). We further document a transfer asymmetry: parallel hybrids exhibit positive cross-task transfer while sequential hybrids suffer catastrophic forgetting. These results establish that hybrid topology fundamentally determines adaptation response, and that component-aware LoRA placement is a necessary design dimension for hybrid architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。