arXiv:2510.22228cs.LGcs.AI2025-10被引 6

剪掉几层模型,推理时的长链条思考能力会崩溃。

When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs

  • 通过测试时扩展机制研究剪枝对长链推理的影响
  • 剪掉1-2层后长链任务性能急剧下降,知识类任务不变
  • 微调无法恢复受损的推理能力,适合重视推理的场景

层剪枝已成为提升大语言模型效率的常用方法。尽管现有方法在通用知识任务上表现良好,但其对长链推理——一种更脆弱却至关重要的能力——的影响尚未被充分探索。本文从测试时扩展(test-time scaling)视角出发,研究层剪枝对长链推理的影响。实验表明,即使剪掉一两层,也会严重损害测试时扩展能力,导致长链推理基准性能急剧下降,而知识密集型和浅层推理任务表现仍稳定。此外,标准监督微调无法恢复已受损的测试时扩展能力。深入分析揭示了该脆弱性的内在机制,指出向推理密集型模型应用层剪枝存在根本风险。研究呼吁重新思考剪枝策略,并为开发保持推理鲁棒性的方法提供洞见。代码已开源。

原文摘要 · Abstract (English)

Layer pruning has emerged as a widely adopted technique for improving the efficiency of large language models (LLMs). Although existing methods demonstrate strong performance retention on general knowledge tasks, their effect on long-chain reasoning, a more brittle yet crucial capability, remains largely unexplored. In this work, we study the impact of layer pruning on long-chain reasoning through the lens of test-time scaling, a key mechanism in modern LLMs that enables strong reasoning capacity by allocating more computation at inference time. With extensive experiments, we demonstrate that pruning even one or two layers can severely impair test-time scaling, with performance collapsing drastically on long reasoning benchmarks even when performance on knowledge-intensive and shallow reasoning tasks remains stable. Furthermore, we find that standard supervised fine-tuning remedies fail to recover test-time scaling once it has deteriorated. Through in-depth analyses, we identify the mechanisms underlying this fragility of test-time scaling and highlight the fundamental risks of applying layer pruning to reasoning-intensive LLMs. These findings call for a rethinking of layer pruning strategies and provide insights for developing methods that preserve the robustness of reasoning. We open-source the codebase in \href{https://github.com/keyu-wang-2002/Layer-Pruning-Harms-Inference-Scaling}{https://github.com/keyu-wang-2002/Layer-Pruning-Harms-Inference-Scaling}.

大模型剪枝推理能力测试时扩展

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。