揭示大模型处理成语时字面与隐喻意义的博弈机制
Tug-of-war between idioms' figurative and literal interpretations in LLMs
- 通过因果追踪发现早期层和注意力头优先提取隐喻义
- 上下文先于成语时，模型会动态修正解释以应对冲突
- 双路径并行保留字面与隐喻两种理解，确保语义可选
成语因其非组合性隐喻含义与字面意义常严重偏离，对语言模型构成独特挑战。本文采用因果追踪方法，系统分析预训练因果变换器如何处理这种歧义。研究发现三种机制：(i) 早期子层及特定注意力头检索成语的隐喻义，同时抑制其字面义；(ii) 当上下文在成语前出现时，模型从最早层利用上下文信息，后期层在上下文与检索结果冲突时进行解释修正；(iii) 随后，选择性竞争路径并行传递两种解释：中间路径优先隐喻义，平行直通路径偏好字面义，从而确保两种解读均被保留。研究为自回归变换器中的成语理解提供了机制层面的证据。
原文摘要 · Abstract (English)
Idioms present a unique challenge for language models due to their non-compositional figurative interpretations, which often strongly diverge from the idiom's literal interpretation. In this paper, we employ causal tracing to systematically analyze how pretrained causal transformers deal with this ambiguity. We localize three mechanisms: (i) Early sublayers and specific attention heads retrieve an idiom's figurative interpretation, while suppressing its literal interpretation. (ii) When disambiguating context precedes the idiom, the model leverages it from the earliest layer and later layers refine the interpretation if the context conflicts with the retrieved interpretation. (iii) Then, selective, competing pathways carry both interpretations: an intermediate pathway prioritizes the figurative interpretation and a parallel direct route favors the literal interpretation, ensuring that both readings remain available. Our findings provide mechanistic evidence for idiom comprehension in autoregressive transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。