模型推理时，隐藏矩阵的秩不影响准确率，说明它没在用多路径推理。
The Gradient Does Not See Rank: Rank-Indifference in Matrix-CODI on ProsQA
- 用矩阵瓶颈压缩推理，通过秩来观察并行路径。
- 不同秩的投影下准确率几乎不变，最高仅差0.6个百分点。
- 即使非线性读出也无视秩，适合研究模型内部表征机制的人看。
连续思维链模型将推理压缩为隐状态令牌。矩阵变体通过d×d矩阵瓶颈处理每个隐状态，使秩成为单样本的结构可观测量。若隐矩阵通过叠加承载并行推理路径，秩应与其相关，降低秩应影响需多步骤任务的准确率。在四个训练设置（三个在ProsQA，一个在低于学习阈值的GSM8K-Aug）中，秩-k投影消融曲线平坦，最大偏差仅0.6个百分点。三组种子复现结果为81.0±2.0百分点，而最终有效秩在{4, 12, 13}之间，损失函数不偏好特定秩。为检验秩盲现象是否仅由扁平化-投影读出引起，训练了四种读出：双线性重参数化、含GELU的双线性+非线性读出、基于SVD并输入奇异值至MLP的读出，以及关于Z Z^T的二次读出。所有四种读出的秩-k曲线均保持平坦（Spearman p值分别为0.63、0.14、0.82、0.46）。非线性读出下曲线仍平坦。对隐矩阵的线性探测器在目标预测上表现劣于原始预训练隐藏状态（AUC 0.673 vs. 0.846）。在普通GPT-2 SFT模型（无矩阵瓶颈、无Z，三组种子，n=500）上进行负向对照，同样干预下秩-k曲线平坦，平均范围0.20个百分点，随机敏感性下限与准确率一致：秩-消融本身混淆了秩无关性与位置无关性。
原文摘要 · Abstract (English)
Continuous chain-of-thought models compress reasoning into latent tokens. Matrix-valued variants, which route each latent token through a d x d matrix bottleneck, introduce rank as a single-sample structural observable on the latent matrix Z. If matrix latents carry parallel reasoning paths via superposition, rank should track them, and truncating Z to low rank should hurt accuracy on tasks whose solutions plausibly require multiple components. Across four training regimes of a matrix-CODI model (three on ProsQA, one on GSM8K-Aug below the learning threshold), the rank-k projection ablation curve is flat to within 0.6 percentage points. A three-seed replication yields 81.0 +/- 2.0 percentage points accuracy while the final effective rank of Z spans {4, 12, 13}; the loss does not reward any particular rank. To test whether rank-blindness arises from the flatten-then-project readout alone, we trained four readouts: a bilinear reparametrization, a bilinear-plus-GELU readout nonlinear in Z, an SVD-augmented readout feeding singular values through an MLP, and a quadratic readout in Z Z^T. All four rank-k curves remain flat (Spearman p-values 0.63, 0.14, 0.82, 0.46). The flat curves persist for readouts nonlinear in Z. A linear probe on Z underperforms a raw pretrained hidden state at target prediction (AUC 0.673 vs. 0.846). A negative control on vanilla GPT-2 SFT (no matrix bottleneck, no Z, three seeds, n=500) reproduces a flat rank-k curve under the same intervention paradigm with pooled-mean range 0.20pp, and a random-h sensitivity floor lands at the same accuracy: the rank-k ablation alone conflates rank-blindness with position-irrelevance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。