提出新方法衡量大模型推理复杂度，并可调控生成文本的计算密度。
Multiple Token Divergence: Measuring and Steering In-Context Computation Density
- 用多令牌差异度量计算复杂性，基于输出分布的KL散度。
- 在数学推理任务中，低差异度对应更高准确率，与难度正相关。
- 无需训练，适配已有模型，适合研究和优化推理过程。
衡量语言模型在上下文中的计算努力是关键挑战，现有指标如下一个词损失无法捕捉推理复杂度。以往基于隐状态可压缩性的方法存在侵入性和不稳定性。本文提出多重令牌差异（MTD），定义为模型完整输出分布与浅层辅助预测头输出分布之间的KL散度，可直接从具有多个预测头的预训练模型中计算，无需额外训练。基于此，我们引入差异引导（Divergence Steering）解码方法，用于控制生成文本的计算特征。实验表明，MTD在区分复杂任务与简单任务方面优于先前方法；在数学推理基准上，MTD与问题难度呈正相关，较低的MTD值关联更高的推理准确性。MTD提供了一种实用、轻量的工具，用于分析与引导语言模型的计算动态。
原文摘要 · Abstract (English)
Measuring the in-context computational effort of language models is a key challenge, as metrics like next-token loss fail to capture reasoning complexity. Prior methods based on latent state compressibility can be invasive and unstable. We propose Multiple Token Divergence (MTD), a simple measure of computational effort defined as the KL divergence between a model's full output distribution and that of a shallow, auxiliary prediction head. MTD can be computed directly from pre-trained models with multiple prediction heads, requiring no additional training. Building on this, we introduce Divergence Steering, a novel decoding method to control the computational character of generated text. We empirically show that MTD is more effective than prior methods at distinguishing complex tasks from simple ones. On mathematical reasoning benchmarks, MTD correlates positively with problem difficulty. Lower MTD is associated with more accurate reasoning. MTD provides a practical, lightweight tool for analyzing and steering the computational dynamics of language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。