arXiv:2410.21332cs.LGcs.AI2024-10ICLR被引 2

提出可抽象序列的非参数模型,实现高效记忆与概念迁移。

Building, Reusing, and Generalizing Abstract Representations from Concrete Sequences

  • 通过分层变量学习,从序列中提取并抽象相似片段为变量。
  • 在语言数据上压缩效率超越LZ算法,且序列似然匹配人类回忆时间。
  • 揭示压缩与泛化间的精确权衡,适合认知建模与小样本学习研究。

人类擅长在不同序列中学习抽象模式,过滤无关细节,并将这些泛化概念迁移到新序列。相比之下,许多序列学习模型缺乏抽象能力,导致内存效率低且迁移性能差。我们提出一种非参数层次化变量学习模型(HVM),能从序列中学习片段,并将上下文相似的片段抽象为变量。HVM在高效组织记忆的同时揭示抽象表示,形成紧凑的序列表征。在babyLM等语言数据集上,HVM学到的词典比标准压缩算法(如Lempel-Ziv)更高效。在需要获取并迁移嵌入序列中的变量的回忆任务中,HVM的序列似然与人类回忆时间高度相关。而大语言模型(LLMs)在传递抽象变量方面远不如人类。通过可调节的抽象层级,我们展示了模型在压缩与泛化之间实现精确权衡。本工作提供了一个捕捉人类认知中抽象表征学习与迁移的认知模型,与现有大模型形成鲜明区别。

原文摘要 · Abstract (English)

Humans excel at learning abstract patterns across different sequences, filtering out irrelevant details, and transferring these generalized concepts to new sequences. In contrast, many sequence learning models lack the ability to abstract, which leads to memory inefficiency and poor transfer. We introduce a non-parametric hierarchical variable learning model (HVM) that learns chunks from sequences and abstracts contextually similar chunks as variables. HVM efficiently organizes memory while uncovering abstractions, leading to compact sequence representations. When learning on language datasets such as babyLM, HVM learns a more efficient dictionary than standard compression algorithms such as Lempel-Ziv. In a sequence recall task requiring the acquisition and transfer of variables embedded in sequences, we demonstrate HVM's sequence likelihood correlates with human recall times. In contrast, large language models (LLMs) struggle to transfer abstract variables as effectively as humans. From HVM's adjustable layer of abstraction, we demonstrate that the model realizes a precise trade-off between compression and generalization. Our work offers a cognitive model that captures the learning and transfer of abstract representations in human cognition and differentiates itself from LLMs.

序列建模抽象表示认知计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。