发现隐式推理模型其实可解释,且正确预测往往对应可还原的推理路径。
Are Latent Reasoning Models Easily Interpretable?
- 通过分析两个前沿模型,发现推理标记常被闲置,未必真正参与决策。
- 在正确预测中,65%-93%能还原出真实推理过程,说明模型行为可解释。
- 提出新方法无需真值即可验证自然语言推理链,适合关注可解释性的研究者。
隐式推理模型(LRMs)因推理成本低且理论上可并行探索多条推理路径而受到广泛关注。然而,其可解释性差,因不使用自然语言进行推理而难以监控。本文研究了两种先进LRM的可解释性:首先,发现逻辑推理数据集上,模型几乎无需使用隐式推理标记即可给出相同答案,这表明推理标记常被冗余使用,可能解释为何其性能未持续优于显式推理方法;其次,当推理标记确实必要时,我们可在正确预测实例中高达65%-93%的概率还原出真实推理轨迹,说明模型通常执行的是预期而非不可解释的推理;最后,提出一种无需先验真值即可从隐式标记中解码可信自然语言推理链的方法,证明对多数正确预测可找到验证过的推理路径,但仅少数错误预测如此。结果表明当前LRM主要编码可解释过程,可解释性本身或可成为预测正确的信号。
原文摘要 · Abstract (English)
Latent reasoning models (LRMs) have attracted significant research interest due to their low inference cost (relative to explicit reasoning models) and theoretical ability to explore multiple reasoning paths in parallel. However, these benefits come at the cost of reduced interpretability: LRMs are difficult to monitor because they do not reason in natural language. This paper presents an investigation into LRM interpretability by examining two state-of-the-art LRMs. First, we find that latent reasoning tokens are often unnecessary for LRMs' predictions; on logical reasoning datasets, LRMs can almost always produce the same final answers without using latent reasoning at all. This underutilization of reasoning tokens may partially explain why LRMs do not consistently outperform explicit reasoning methods and raises doubts about the stated role of these tokens in prior work. Second, we demonstrate that when latent reasoning tokens are necessary for performance, we can decode gold reasoning traces up to 65-93% of the time for correctly predicted instances. This suggests LRMs often implement the expected solution rather than an uninterpretable reasoning process. Finally, we present a method to decode a verified natural language reasoning trace from latent tokens without knowing a gold reasoning trace a priori, demonstrating that it is possible to find a verified trace for a majority of correct predictions but only a minority of incorrect predictions. Our findings highlight that current LRMs largely encode interpretable processes, and interpretability itself can be a signal of prediction correctness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。