通过混合控制机制提升Transformer微调的表示能力
Mixture-of-Control: State-Aware Fine-Tuning for Transformer-based Models
- 将每层控制状态视为专家，动态融合局部与全局信息
- 在保持低内存开销下，跨块通信显著提升模型表现
- 适合需要高效微调的NLP和视觉任务场景
基于状态的微调已成为Transformer模型参数高效适配的有力替代方案，通过更新轻量级控制状态而非模型权重，在大幅节省内存的同时保持参数效率。然而，现有方法多仅在单层内更新控制信号，限制了跨层信息交互，制约了表征能力。同时，支持跨层通信的机制常伴随显著计算开销，影响实际应用。本文提出Mixture-of-Control（MoC），一种轻量级微调框架，通过将块级控制状态作为稀疏专家混合系统中的专家，实现高效跨块通信，自适应融合局部与全局控制信号以增强表示学习。在多种Transformer基准上的实验证明，MoC在保持与现有方法相当的内存与计算效率的同时，优于当前基于状态的方法。
原文摘要 · Abstract (English)
State-based fine-tuning has emerged as a compelling alternative to weight-based adaptation for transformers, updating lightweight controls into states rather than model weights, offering substantial memory savings while retaining parameter efficiency. However, most existing state-based methods typically apply only per-block control updates, which limits inter-block information exchange and restricts representational adaptation. Meanwhile, prior mechanisms that enable cross-block communication often introduce considerable computational overhead, reducing their practicality for efficient fine-tuning. We introduce Mixture-of-Control (MoC), a lightweight fine-tuning framework that adaptively integrates local and global control signals to enhance representation learning. MoC treats block-wise control states as experts in a sparse mixture-of-experts process, enabling efficient communication across transformer blocks. Empirical results across diverse transformer-based benchmarks demonstrate that MoC outperforms state-based methods while maintaining a comparable memory and computational efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。