对比人类与大模型在答题中的思维结构，发现大模型推理机制难被人类理解。
Interpretable Humans, Alien LLMs: Expert Analysis of Latent Structures in Assessment Responses
- 用探索性因子分析分离人类和大模型的潜在思维模式
- 专家能解读70%以上人类因子，仅能解读一半大模型化学因子
- 揭示大模型推理机制与人类认知存在本质差异，适合评估可信度研究者
大语言模型（LLMs）的评估高度依赖人工设计的测试，隐含假设人工智能与人类使用相似的认知结构。我们挑战这一假设，探究大模型表现背后的潜在因素是否具备与人类学习者相同的可解释意义。通过分析人类及六种大模型在数量推理和化学测试中的作答，分别进行探索性因子分析（EFA），再由学科专家盲评生成的因子图以赋予教学意义。专家成功解读了多数人类因子；但在数量推理中完全无法解释大模型因子，在化学任务中仅能解读一半。该框架结合数据驱动分析与专家盲评，表明大模型常基于统计上不透明的机制运行，其推理方式显著区别于人类认知。
原文摘要 · Abstract (English)
The evaluation of large language models (LLMs) relies heavily on human-designed assessments, implicitly assuming that AI and humans employ similar underlying cognitive constructs. Challenging this assumption, we investigate whether the latent factors governing LLM performance carry the same substantive, human-interpretable meaning as the cognitive constructs governing human learners. Using responses from humans and six LLMs across quantitative reasoning and chemistry assessments, we conducted Exploratory Factor Analysis (EFA) separately for both groups. Subject-Matter Experts (SMEs) then blindly evaluated the resulting factor graphs to ascribe pedagogical meaning to the emerged constructs. SMEs successfully interpreted most of the human-derived factors. Conversely, they could not ascribe meaning to any LLM-derived factors in quantitative reasoning and interpreted only half of the LLM factors in chemistry. By combining data-driven EFA with blind expert interpretation, this framework shows that LLMs frequently operate on statistically opaque mechanisms distinct from human reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。