arXiv:2606.05346cs.CL2026-06被引 2

用语言模型隐状态轨迹预测人类阅读成本,比传统方法更准。

Trajectory Dynamics in Language Model Hidden States Predict Human Processing Costs Beyond Surprisal

论文配图:Trajectory Dynamics in Language Model Hidden States Predict Human Processing Costs Beyond Surprisal
图 1 · 摘自论文原文
  • 通过拟合模型隐藏状态的线性轨迹,计算偏离程度来衡量语义演进动量。
  • 在自然故事语料上,该指标独立预测阅读时长,尤其在歧义句中效果显著。
  • 适用于研究语言理解机制、认知计算建模及大模型语义动态分析者。

人类语言理解是逐步进行的:每个词都在前文语境中被处理,意义随时间渐进构建。目前主流的增量加工成本预测指标是“意外度”(surprisal),即给定上下文下某词的负对数概率。但意外度将丰富的序列表征简化为单个标量,丢失了意义演进方向的信息。动力系统视角认为,语义状态的演变轨迹本身应影响加工成本,语言可能具有局部动量——说话者会提前规划接下来几个词。本文提出“轨迹外推误差”:在每个词处,用Transformer语言模型的前序隐藏状态拟合线性轨迹,并测量实际状态与外推路径的偏差。在Natural Stories语料上,该指标与意外度几乎正交(r = .044),且独立预测自控阅读时间。该效应在花园路径句中尤为明显,随模型规模提升(从GPT-2 Small到Large),并在不同架构(GPT-2 vs. Pythia/RoPE)间复现。位移控制实验表明,该效应不能归结为表征变化幅度:位移与外推误差预测方向相反。结果揭示加工成本包含两个可分离成分:词级预测误差(意外度)和对语义演进动量的敏感性(轨迹外推误差)。

原文摘要 · Abstract (English)

Human language comprehension unfolds sequentially: each word is processed in the context of those that came before, and the interpretation builds incrementally over time. Surprisal, the negative log probability of a word given its context, has been the dominant predictor of incremental processing cost. But surprisal reduces rich sequential representations to a single scalar at each word, discarding information about the direction in which the interpretation has been evolving. Dynamical-systems approaches suggest that the trajectory of the evolving interpretive state, not just its position at each moment,should shape processing, and language itself may have local momentum, since speakers plan utterances a few words at a time. We introduce trajectory extrapolation error: at each word, we fit a linear trajectory to the preceding hidden states of a transformer language model and measure deviation from the extrapolated path. On the Natural Stories corpus, this measure is nearly orthogonal to surprisal (r = .044) and independently predicts self-paced reading times. The effect is especially pronounced in garden-path sentences, strengthens with model scale (GPT-2 Small to Large), and replicates across architectures with different positional encoding schemes (GPT-2 vs. Pythia/RoPE). A displacement control shows the effect is not reducible to representational change magnitude: displacement and extrapolation error predict in opposite directions. These findings reveal two dissociable components of processing cost: word-level prediction error (surprisal) and sensitivity to the local momentum of the unfolding interpretation (trajectory extrapolation error).

语言理解认知建模轨迹分析大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。