针对模拟存内计算硬件中的误差，提出混合分解方法提升状态空间模型鲁棒性。
HPD: Hybrid Projection Decomposition for Robust State Space Models on Analog CIM Hardware
- 将输出投影层权重分解为奇异值与数字校正部分，适配硬件并抑制误差
- 在多种噪声下使困惑度降低99.57%，常识推理任务准确率提升96.67%
- 适合部署于模拟存内计算硬件的长序列模型优化，尤其关注最后层稳定性
状态空间模型（SSMs）是处理长序列的高效替代方案，其对矩阵乘法的依赖使其适合存内计算（CIM）架构，从而提升能效。然而，CIM中器件非理想性会引入权重扰动，影响推理精度。本文系统分析了SSMs在噪声条件下的鲁棒性，发现最后一层块和输出投影层更易受扰动影响。基于此，我们提出HPD——一种针对输出投影层的混合投影分解策略：将原始权重矩阵通过SVD分解为U和Σ，以适配现有硬件；同时将V^T移至数字硬件，实现精确且鲁棒的修正。在Mamba模型上的全面测试表明，该方法在不同噪声条件下可使困惑度降低高达99.57%，在PIQA常识推理基准上准确率提升达96.67%。
原文摘要 · Abstract (English)
State Space Models (SSMs) are efficient alternatives to traditional sequence models, excelling at processing long sequences with lower computational complexity. Their reliance on matrix multiplications makes them ideal for compute-in-memory (CIM) architectures, which improve energy efficiency by computing within memory arrays. However, device non-idealities in CIM introduce weight perturbations that can degrade inference accuracy. In this paper, we systematically analyze the robustness of SSMs under noisy conditions, identifying that the final block and output projection layers are more susceptible to perturbations compared to other components. Building on these insights, we propose HPD, a Hybrid Projection Decomposition strategy for the last output projection layer. We replace the original weight matrix with the multiplication of U and Σ in its SVD to ensure compatibility with existing hardware architectures, while offloading V> to digital hardware for precise and robust correction. Comprehensive tests on Mamba models show that our method reduces perplexity by up to 99.57% under various noise conditions compared to baseline models, with accuracy gains of up to 96.67% on the PIQA benchmark for commonsense reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。