通过低秩结构蒸馏，让大模型内部推理更高效准确。
LoRi: Low-Rank Distillation for Implicit Reasoning
- 用低秩张量空间对齐师生模型的推理轨迹
- 在数学推理任务中逼近显式链式思维表现
- 适合需要高效内部推理的模型优化场景
隐式思维链（iCoT）方法旨在将推理过程内化到大语言模型中，但通常表现不如显式链式思维提示。我们发现隐藏状态的推理轨迹具有低秩结构。基于此，提出一种低秩蒸馏框架，通过一阶和二阶统计量对齐教师与学生模型的推理轨迹，共享一个低秩张量子空间。该方法捕捉了推理的全局结构，同时支持紧凑的潜在推理过程。我们在多个模型家族（包括LLaMA和Qwen）的不同规模上，于数学推理基准上评估该方法，结果一致提升性能，尤其在复杂多步任务中表现显著，接近显式链式思维精度，并优于以往iCoT蒸馏方法。
原文摘要 · Abstract (English)
Implicit chain-of-thought (iCoT) methods aim to internalize reasoning in large language models, but often underperform explicit CoT prompting. We empirically find that hidden-state reasoning trajectories exhibit low-rank structure. Motivated by this observation, we propose a low-rank distillation framework that transfers reasoning by aligning teacher and student trajectories in a shared low-rank tensor subspace using first- and second-order statistics. The resulting formulation captures the global structure of reasoning while supporting a compact latent reasoning process. We evaluate the method across multiple model families, including LLaMA and Qwen, at different scales on mathematical reasoning benchmarks. Our approach consistently improves performance, especially on challenging multi-step tasks, approaching explicit CoT accuracy and outperforming prior iCoT distillation methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。