研究特征相关性如何影响记忆模型的存储能力，发现相关性会轻微降低容量。
Effects of Feature Correlations on Associative Memory Capacity
- 通过构造不同特征相关性和模式分离度的数据集，分析容量变化
- 容量随输入空间分离度呈指数增长，相关性小幅降低容量
- 对高阶特征交互建模能力有限，适合关注数据结构影响的研究者
我们研究特征相关性对密集关联记忆（DAM）模型存储容量的影响。实际机器学习中数据常具有特征相关性且在输入空间中学习表示，但现有容量分析未考虑此因素。为此，我们构建了一个实证框架，系统生成基于汉明距离（Hamming distance）的、具有不同特征相关性和模式分离度的数据集，并使用简单的二分搜索算法计算模型的存储容量。实验表明，记忆容量随输入空间分离度呈指数级增长；特征相关性虽不改变这一关系的本质，但在固定分离度下会略微降低容量。该效应在能量函数的高阶多项式程度下更显著，表明关联记忆模型在刻画特征间高阶交互时能力受限。研究结果连接了理论分析与实际场景，可能启发更注重数据结构的方法。
原文摘要 · Abstract (English)
We investigate how feature correlations influence the capacity of Dense Associative Memory (DAM), a Transformer attention-like model. Practical machine learning scenarios involve feature-correlated data and learn representations in the input space, but current capacity analyses do not account for this. We develop an empirical framework to analyze the effects of data structure on capacity dynamics. Specifically, we systematically construct datasets that vary in feature correlation and pattern separation using Hamming distance from information theory, and compute the model's corresponding storage capacity using a simple binary search algorithm. Our experiments confirm that memory capacity scales exponentially with increasing separation in the input space. Feature correlations do not alter this relationship fundamentally, but reduce capacity slightly at constant separation. This effect is amplified at higher polynomial degrees in the energy function, suggesting that Associative Memory is more limited in depicting higher-order interactions between features than patterns. Our findings bridge theoretical work and practical settings for DAM, and might inspire more data-centric methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。