压缩递归推理模型时,局部预测保留但全局推理失效,需新方法应对。
What Survives When You Compress a Recursive Reasoner for the Edge?
- 发现量化误差在递归循环中累积,破坏整体推理能力。
- INT4压缩下任务精确率降至零,但局部准确率仍保持。
- 提出无标签的轨迹保真度指标,可提前预测压缩损伤与恢复效果。
递归推理模型通过反复更新隐状态,仅用数百万参数即可解决复杂结构化任务。部署于边缘设备需大幅压缩,但与传统序列模型不同,量化误差在递归循环中累积而非输出词元之间,导致常规压缩直觉失效。本文在全精度扫描、三类任务和两种架构下研究压缩对模型的影响,发现激进压缩虽保留局部预测能力,却使全局推理完全崩溃:在非训练感知的INT4剪枝、蒸馏及线性注意力下,谜题精确率归零。词级目标(如量化感知训练)无法修复该问题。损伤具有架构特性——仅影响MLP混合型递归,不影响同一任务上的注意力机制。通过无需重训的逐通道校准INT4,可逆转该现象。引入‘携带轨迹保真度’(carry-trajectory fidelity),即与全精度推理路径的余弦相似度,作为无标签信号,能预判损伤及其恢复。综合结果形成部署方案:流式嵌入消除99.4MB瓶颈,单周期INT8实现6倍更少计算量(8MB SoC)且达到全深度精度,校准后INT4可部署于4MB微控制器。
原文摘要 · Abstract (English)
Recursive reasoning models can solve complex structured tasks with only a few million parameters by repeatedly updating a latent state. Deploying these models on edge hardware requires significant compression, but unlike conventional sequence models, quantization errors compound across recursive reasoning cycles rather than across output tokens. As a result, standard intuitions about compression fail to apply. In this work, we ask what survives when recursive reasoners are compressed. Across a full precision sweep, three tasks, and two recursive architectures, we find that aggressive compression preserves local prediction but destroys global reasoning: cell accuracy holds while puzzle-exact accuracy collapses to zero under naive INT4 pruning, distillation, and linear attention alike. Token-level objectives, including quantization-aware training, cannot repair it. The collapse is architectural -- it strikes MLP-mixing recursion but not attention on the same task -- and we reverse it with per-channel calibrated INT4 without retraining. We also introduce carry-trajectory fidelity, the cosine similarity to the full-precision reasoning path, as a label-free signal that predicts this damage and its recovery before a task evaluation. The combined result is a deployment recipe: flash-streamed embeddings remove a 99.4MB bottleneck, INT8 at one cycle matches full-depth accuracy at 6x fewer FLOPs (8MB SoC), and calibrated INT4 fits a 4MB microcontroller.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。