给大模型的知识分类打标，看清它到底知道什么、多确定。
Decoding Knowledge in Large Language Models: A Framework for Categorization and Comprehension
- 按准确性和置信度将知识分六类，突破对错二元评价
- 思维链提示让模型更自信且更准，尤其在对齐后模型中效果翻倍
- 高层网络存高置信度知识，中低层易出不确定认知
理解大语言模型如何获取、保留和应用知识仍是开放挑战。本文提出新框架K-(CSA)^2，从正确性与置信度两个维度对模型知识进行分类，定义六类知识，涵盖从高度自信的正确到自信错误的认知，实现对模型理解力的精细评估，超越传统二元准确率。基于该框架，我们发现思维链提示（CoT）和基于人类反馈的强化学习可显著改变模型内部（预训练）与外部（上下文依赖）知识结构。CoT特别提升基础模型性能，并在对齐模型上呈现协同增益。此外，层级分析显示，模型高层编码更多高置信度知识，而低置信度知识多集中于中下层。
原文摘要 · Abstract (English)
Understanding how large language models (LLMs) acquire, retain, and apply knowledge remains an open challenge. This paper introduces a novel framework, K-(CSA)^2, which categorizes LLM knowledge along two dimensions: correctness and confidence. The framework defines six categories of knowledge, ranging from highly confident correctness to confidently held misconceptions, enabling a nuanced evaluation of model comprehension beyond binary accuracy. Using this framework, we demonstrate how techniques like chain-of-thought prompting and reinforcement learning with human feedback fundamentally alter the knowledge structures of internal (pre-trained) and external (context-dependent) knowledge in LLMs. CoT particularly enhances base model performance and shows synergistic benefits when applied to aligned LLMs. Moreover, our layer-wise analysis reveals that higher layers in LLMs encode more high-confidence knowledge, while low-confidence knowledge tends to emerge in middle-to-lower layers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。