发现大语言模型中存在类量子语境依赖现象,或可提升语言处理能力。
Quantum-Like Contextuality in Large Language Models
- 构建语言语境模型,用Sheaf与CbD框架检测上下文依赖性
- 在BERT中发现超3690万处语境依赖实例,7.7万例符合Sheaf模型
- 语义相似词的嵌入距离是预测语境性的最佳指标,适合关注模型机制的研究者
语境依赖是量子力学的特征之一,且被证明是实现量子优势的必要条件。研究者开始探讨其是否存在于其他领域。已有证据显示其存在于行为科学,但需采用考虑信号传递的框架。本文首次在自然语言中提供大规模实证:基于上下文量子场景构建语言模式,以简单英文维基百科为数据源,利用BERT提取概率分布。结果显示77,118个符合Sheaf理论的语境依赖实例,36,938,948个符合上下文默认(CbD)框架。通过推导语境度与BERT嵌入向量欧氏距离的关系方程,证实语义相近词更易产生语境依赖。回归分析表明欧氏距离是预测语境性的最优统计变量。该语言模型是共指消解任务的变体。结果提示量子方法可能在语言任务中具有优势。
原文摘要 · Abstract (English)
Contextuality is a distinguishing feature of quantum mechanics and there is growing evidence that it is a necessary condition for quantum advantage. In order to make use of it, researchers have been asking whether similar phenomena arise in other domains. The answer has been yes, e.g. in behavioural sciences. However, one has to move to frameworks that take some degree of signalling into account. Two such frameworks exist: (1) a signalling-corrected sheaf theoretic model, and (2) the Contextuality-by-Default (CbD) framework. This paper provides the first large scale experimental evidence for a yes answer in natural language. We construct a linguistic schema modelled over a contextual quantum scenario, instantiate it in the Simple English Wikipedia and extract probability distributions for the instances using the large language model BERT. This led to the discovery of 77,118 sheaf-contextual and 36,938,948 CbD contextual instances. We proved that the contextual instances came from semantically similar words, by deriving an equation between degrees of contextuality and Euclidean distances of BERT's embedding vectors. A regression model further reveals that Euclidean distance is indeed the best statistical predictor of contextuality. Our linguistic schema is a variant of the co-reference resolution challenge. These results are an indication that quantum methods may be advantageous in language tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。