受联想记忆启发，新架构让模型更快学会上下文信息。
Associative memory inspires improvements for in-context learning using a novel attention residual stream architecture
- 借鉴生物记忆机制，设计注意力残差流结构。
- 在小模型上验证，上下文学习能力提升且收敛更快。
- 适合研究高效小模型与类脑计算的读者。
大语言模型（LLMs）具备利用输入序列中上下文信息回答训练中未见数据的能力，称为上下文学习（ICL）。人类和非人类动物也具有类似能力，但其神经架构与LLMs差异显著。尽管如此，LLMs中的注意力机制与现代联想记忆模型高度相似，后者广泛用于建模生物记忆系统。受此启发，我们提出一种能实现ICL的联想记忆模型，并据此设计了一种新型残差流架构，使信息可直接在注意力头间流动。我们在两层Transformer中测试该架构，发现其在训练过程中能更快速地展现ICL能力。进一步将该架构应用于800万和10亿参数的小型语言模型，聚焦注意力头值，结果表明在更大、更自然的规模下仍表现更优。
原文摘要 · Abstract (English)
Large language models (LLMs) demonstrate an impressive ability to utilise information within the context of their input sequences to appropriately respond to data unseen by the LLM during its training procedure. This ability is known as in-context learning (ICL). Humans and non-human animals demonstrate similar abilities, however their neural architectures differ substantially from LLMs. Despite this, a critical component within LLMs, the attention mechanism, resembles modern associative memory models, widely used in and influenced by the computational neuroscience community to model biological memory systems. Using this connection, we introduce an associative memory model capable of performing ICL. We use this as inspiration for a novel residual stream architecture which allows information to directly flow between attention heads. We test this architecture during training within a two-layer Transformer and show its ICL abilities manifest more quickly than without this modification. We then apply our architecture in small language models with 8 million and 1 billion parameters, focusing on attention head values, with results also indicating improved performance at these larger and more naturalistic scales.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。