arXiv:2604.26355cs.CL2026-04中稿 · COLM被引 1

用熵引导的超令牌压缩大模型推理,省计算还可解释。

Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokens

  • 从推理文本中提取高频结构令牌,生成压缩超令牌。
  • 平均缩短推理长度8.1%,多数场景精度不变或不确定。
  • 压缩后仍能识别回溯、验证等可解释推理模式。

大型语言模型的推理过程消耗大量计算资源,但其分词级信息结构尚未被充分研究。我们发现推理分词可分为两类:低熵结构令牌(重复出现的框架性短语)和高熵有机令牌(问题相关的具体内容)。这一不对称性启发了一个简单且与模型无关的压缩流程:对模型自身的推理轨迹应用跨词BPE合并,生成捕捉常见结构模式的超令牌,并通过监督微调使模型学会使用这些超令牌。在三个模型家族和五个数学推理基准上,该方法平均将推理轨迹缩短8.1%;在TOST等效性分析(±2个百分点容差)下,13/15个模型-基准组合的准确率等效或结果不确定(主要为AIME在N=30时),仅2个组合出现非等效下降(DeepSeek-R1-Distill-Llama-70B在MATH-500和OlympiadBench)。此外,学习到的超令牌常对应可解释的推理行为,如回溯、验证和策略转换。由此可进行紧凑的推理轨迹结构分析:正确轨迹表现出更多恢复与验证模式,错误轨迹则更多重复犹豫与未解决的反论。完整代码已开源。

原文摘要 · Abstract (English)

Reasoning in Large Language Models incurs significant inference-time compute, yet the token-level information structure of reasoning traces remains underexplored. We observe that reasoning tokens split into two functional types: low-entropy structural tokens (recurring phrases that scaffold the reasoning process) and higher-entropy organic tokens (problem-specific content that drives toward a solution). This asymmetry motivates a simple, model-agnostic compression pipeline: apply cross-word BPE merges on a model's own reasoning traces to derive \textit{supertokens} that capture frequent structural patterns, then teach the model to adopt them via supervised fine-tuning. Across three model families and five mathematical reasoning benchmarks, our approach shortens reasoning traces by 8.1% on average; under a TOST equivalence analysis at a +/- 2pp margin, accuracy is equivalent or inconclusive on 13/15 model -- benchmark cells (2 pass equivalence, 11 inconclusive, predominantly AIME at N=30, with non-equivalent degradation on 2/15 cells (DeepSeek-R1-Distill-Llama-70B on MATH-500 and OlympiadBench). Beyond compression, learned supertokens often align with interpretable reasoning moves such as backtracking, verification, and strategy shifts. This enables a compact structural analysis of reasoning traces: correct traces show more recovery and verification patterns, while incorrect traces show more repeated hedging and unresolved counterarguments. We release the full pipeline as open-source code.

大模型压缩推理优化可解释性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。