通过微调循环层初始状态，实现零推理开销的模型适配，性能超越LoRA。
S0 Tuning: Zero-Overhead Adaptation of Hybrid Recurrent-Attention Models
- 仅优化每层循环的初始状态矩阵，冻结所有参数，无推理延迟。
- 在HumanEval上比LoRA高10.8个百分点，Qwen3.5-4B上贪心通过率提升23.6个百分点。
- 适配文件仅48MB，任务切换无需权重合并或重载模型，适合资源受限场景。
基于约48个经执行验证的HumanEval训练解法，每层循环结构仅微调一个初始状态矩阵，在不增加推理开销的前提下，使HumanEval得分优于LoRA 10.8个百分点（p < 0.001）。该方法称为S0 tuning，仅优化每层循环的初始状态矩阵，其余模型权重保持冻结。在Qwen3.5-4B（GatedDeltaNet混合模型）上，贪心通过率提升23.6 ± 1.7个百分点（10次种子实验）。在FalconH1-7B（Mamba-2混合模型）上，S0达到71.8% ± 1.3，而LoRA为71.4% ± 2.4（3次种子），差异不显著，且无需权重合并。跨领域迁移在MATH-500上提升4.8个百分点（p = 0.00002，8次种子），GSM8K上提升2.8个百分点（p = 0.0003，10次种子）；文本转SQL基准Spider无迁移效果，符合轨迹引导机制预期。纯Transformer模型（Qwen2.5-3B）的前缀微调在全部9种配置下性能下降13.9个百分点。在Qwen3.5上，每步状态偏移变体达+27.1个百分点，高于S0与LoRA，但带来每步推理成本。结果表明，在缺乏验证监督时，循环状态初始化是混合语言模型的强大零推理开销参数高效微调表面。微调状态文件约48MB，任务切换无需权重合并或模型重载。代码与库：https://github.com/jackyoung27/s0-tuning。
原文摘要 · Abstract (English)
Using roughly 48 execution-verified HumanEval training solutions, tuning a single initial state matrix per recurrent layer, with zero inference overhead, outperforms LoRA by +10.8 pp (p < 0.001) on HumanEval. The method, which we call S0 tuning, optimizes one state matrix per recurrent layer while freezing all model weights. On Qwen3.5-4B (GatedDeltaNet hybrid), S0 tuning improves greedy pass@1 by +23.6 +/- 1.7 pp (10 seeds). On FalconH1-7B (Mamba-2 hybrid), S0 reaches 71.8% +/- 1.3 and LoRA reaches 71.4% +/- 2.4 (3 seeds), statistically indistinguishable at this sample size while requiring no weight merging. Cross-domain transfer is significant on MATH-500 (+4.8 pp, p = 0.00002, 8 seeds) and GSM8K (+2.8 pp, p = 0.0003, 10 seeds); a text-to-SQL benchmark (Spider) shows no transfer, consistent with the trajectory-steering mechanism. A prefix-tuning control on a pure Transformer (Qwen2.5-3B) degrades performance by -13.9 pp under all nine configurations tested. On Qwen3.5, a per-step state-offset variant reaches +27.1 pp, above both S0 and LoRA but with per-step inference cost. Taken together, the results show that recurrent state initialization is a strong zero-inference-overhead PEFT surface for hybrid language models when verified supervision is scarce. The tuned state is a ~48 MB file; task switching requires no weight merging or model reload. Code and library: https://github.com/jackyoung27/s0-tuning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。