一个模型训练一次,就能按需动态缩放,适配不同设备性能。
Elastic Spectral State Space Models for Budgeted Inference
- 基于频谱滤波的态空间模型,通过轻量门控实现运行时灵活裁剪。
- 单个模型在不同参数规模下表现接近主流Transformer与SSM。
- 支持细粒度资源调控,适合边缘计算等资源受限场景。
基础模型通常以固定算力训练,但实际应用需适配多样化的设备资源。现有方法依赖训练多个模型变体或模型蒸馏,需额外训练且仅支持预设尺寸,难以实现运行时细粒度调整。本文提出弹性频谱态空间模型(ES-SSM),仅需一次全规模训练,即可直接裁剪至任意规模用于预算内推理,无需重训练。该模型基于态空间模型的汉克尔频谱滤波,结合轻量级输入自适应门控,在随机频谱预算下训练。通过共享掩码归一化规则作用于有序频谱通道,促使预测能力集中于低索引分量,高索引分量主要起精修作用。我们在涵盖文本、逻辑、检索、视觉和音频的长序列基准上验证了该算法。结果表明,单个训练好的ES-SSM模型经裁剪后,在相似参数规模下性能可媲美现代Transformer与SSM基线。此外,在多种运行时预算下测试,观察到在广泛裁剪水平下性能曲线平滑稳定。
原文摘要 · Abstract (English)
Foundation models are typically trained at a fixed computational capacity, while real-world applications require deployment across platforms with different resource constraints. Current approaches usually rely on training families of model variants or model distillation, which requires additional training and supports only a pre-selected set of sizes rather than fine-grained adaptation at runtime. In this paper, we propose Elastic Spectral State Space Models (ES-SSM), which require only one-time training at full capacity, but can be directly truncated into arbitrary scales for budgeted, runtime inference without retraining. Our ES-SSM builds on Hankel spectral filtering over a state space model (SSM), coupled with a lightweight input-adaptive gate trained under randomized spectral budgets. Using a shared masked normalization rule over the ordered spectral channels, we encourage predictive capability to concentrate in low-index components, while higher-index components act primarily as refinement. We test our algorithm across long-sequence benchmarks spanning text, logic, retrieval, vision, and audio. We demonstrate that a single ES-SSM model trained once can be truncated to provide competitive performance compared with modern Transformer and SSM baselines at similar parameter scales. Furthermore, by testing under various runtime budgets, we observe smooth and stable budget-performance curves over a wide range of truncation levels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。