用哈密顿降阶建模提升长序列微调效果，适配器位置决定性能优劣。
SSM Adapters via Hankel Reduced-order Modeling: Injection Site Determines Task Suitability in Long-Context Fine-Tuning
- 基于哈密顿矩阵降阶构建新型SSM适配器，支持快速并行计算。
- 在840万参数下，对长文本任务准确率提升超34%，生成质量提升71%。
- 适合需要状态累积的长序列任务，替代传统低秩微调方案。
尽管参数高效微调（PEFT）通常作用于注意力投影层，但其在需序列状态积累的任务中仍缺乏研究。本文探讨了将状态空间模型（SSMs）适配器应用于此类任务的可能性，并比较了在MLP模块注入的效果。提出基于哈密顿降阶建模（HRM）的适配器，通过经验哈密顿格拉姆的平衡截断初始化，利用系统矩阵$ar{A}$的时间不变性，实现精确的快速傅里叶变换并行扫描，在所有上下文长度下计算效率与LoRA相当。在Mistral-7B（840万可训练参数）上的等参数评估显示，HRM在LongBench任务中优于各类LoRA变体，包括QuALITY（相对准确率提升34.8%）和QMSum（相对ROUGE-1提升71.6%）。此外，HRM在18种合成状态追踪任务（如DFA、Parity）及字符级语言建模（enwik8）中也表现出一致优势。门控分析表明，HRM能有效学习调控递归行为，为长序列建模提供稳健的架构替代方案。
原文摘要 · Abstract (English)
While parameter-efficient fine-tuning (PEFT) typically targets attention projectors, its efficacy for tasks requiring sequential state accumulation remains under-explored. We examine if PEFT for such tasks can benefit from state space model (SSMs) adapters, and if MLP blocks are better injection sites. We introduce Hankel Reduced order Model (HRM) adapter, an SSM-based residual module initialized via Balanced Truncation of empirical Hankel Grammians. By leveraging the time-invariance of the system matrix $\bar{A}$, HRM enables an exact FFT-based parallel scan, achieving computational parity with LoRA across all context lengths. In iso-parametric evaluations on Mistral-7B (8.4M trainable parameters), HRM outperforms LoRA variants on LongBench tasks, including QuALITY (+34.8\% relative accuracy) and QMSum (+71.6\% relative ROUGE-1). HRM further demonstrates consistent superiority across 18 configurations of synthetic state-tracking (DFA, Parity) and character-level language modeling (enwik8). Gate analysis reveals that HRM adapters effectively learn to modulate recurrence, providing a robust architectural alternative to low-rank adaptation for long-context sequence modeling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。