通过动态低秩追踪，让递归模型节省60%激活内存而不丢精度。
LASER: Low-Rank Activation SVD for Efficient Recursion
- 用功率迭代动态追踪激活主方向，构建可演化的低秩基。
- 实现约60%激活内存压缩，准确率无显著下降。
- 适合研究递归模型效率与隐式推理机制的学者。
递归架构如微型递归模型（TRMs）通过迭代隐式计算进行推理，但其推理轨迹的几何结构仍不清晰。我们分析了TRMs在递归展开过程中的激活流形，发现激活占据一个有效线性、低维子空间，其主方向可通过低成本功率迭代动态追踪。这表明权重共享使迭代计算集中在少数主导特征方向上，且该集中程度在不同计算位置差异明显。为此提出LASER（低秩激活SVD高效递归），一种基于无矩阵子空间追踪与保真度触发重置机制的动态压缩框架，在不造成统计显著精度损失的前提下，实现约60%的激活内存节省。我们的分析引发对递归架构如何分配表征容量于隐式推理的思考，并探讨能否利用此集中特性提升潜在计算的效率与稳定性。
原文摘要 · Abstract (English)
Recursive architectures such as Tiny Recursive Models (TRMs) perform implicit reasoning through iterative latent computation, yet the geometric structure of these reasoning trajectories remains poorly understood. We investigate the activation manifold of TRMs during recursive unrolling and find that activations occupy an effectively linear, low-dimensional subspace whose principal directions can be tracked dynamically with cheap power iterations. This suggests that weight-sharing concentrates iterative computation along a small number of dominant eigendirections, and we find that this concentration varies sharply across computational sites. We exploit this structure through LASER (Low-Rank Activation SVD for Efficient Recursion), a dynamic compression framework that maintains an evolving low-rank basis via matrix-free subspace tracking with a fidelity-triggered reset mechanism, achieving ${\sim}60\%$ activation memory savings with no statistically significant accuracy degradation. Our analysis raises questions about how recursive architectures allocate representational capacity during implicit reasoning, and whether this concentration can be exploited to improve the efficiency and stability of latent computation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。