通过分析模型内部激活值，可提前预测答案对错并判断外部上下文有效性。
LLM Microscope: What Model Internals Reveal About Answer Correctness and Context Utilization
- 用首词激活值训练分类器，75%准确率预判输出正确性。
- 新指标比提示方法更有效区分正确与错误上下文。
- 适合想提升LLM可信度的研究者和开发者使用。
尽管大语言模型（LLMs）应用广泛，其可信度仍是关键挑战：模型常以高置信度生成错误信息。虽然上下文有助于引导生成，但识别查询是否需检索上下文以及评估上下文有效性仍具难度。本文将可解释性方法具体化，探究仅从模型激活值能否预测输出正确性，并探索模型内部是否蕴含外部上下文效用的信号。我们区分正确、错误及无关上下文，引入量化指标。在六种不同模型上的实验表明，仅基于首个输出词的中间层激活值训练的简单分类器，即可实现约75%的输出正确性预测准确率，支持早期审计。基于模型内部的指标显著优于提示基线，在区分正确与错误上下文方面表现更优，能有效防范污染上下文带来的误差。这些发现为理解大模型决策机制提供了新视角。代码已公开于 https://github.com/jiarui-liu/LLM-Microscope。
原文摘要 · Abstract (English)
Although large language models (LLMs) have tremendous utility, trustworthiness is still a chief concern: models often generate incorrect information with high confidence. While contextual information can help guide generation, identifying when a query would benefit from retrieved context and assessing the effectiveness of that context remains challenging. In this work, we operationalize interpretability methods to ascertain whether we can predict the correctness of model outputs from the model's activations alone. We also explore whether model internals contain signals about the efficacy of external context. We consider correct, incorrect, and irrelevant context and introduce metrics to distinguish amongst them. Experiments on six different models reveal that a simple classifier trained on intermediate layer activations of the first output token can predict output correctness with about 75% accuracy, enabling early auditing. Our model-internals-based metric significantly outperforms prompting baselines at distinguishing between correct and incorrect context, guarding against inaccuracies introduced by polluted context. These findings offer a lens to better understand the underlying decision-making processes of LLMs. Our code is publicly available at https://github.com/jiarui-liu/LLM-Microscope
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。