让AI像人一样记忆图像，用可重构的内存系统提升多模态理解能力
Do You Remember? Toward Memory-Centric Multimodal AI
- 设计三阶段架构：先压缩图像为离散令牌，再联合视觉与文本编码，最后用扩散模型重建图像
- 发现大模型隐藏状态几乎无法恢复图像，而引入记忆矩阵后重建质量接近最优压缩上限
- 通过局部动态更新机制解决梯度消失问题，实现高效、低损耗的记忆存储与回放
人类记忆具有重构性而非忠实记录。现有多模态大模型（MLLM）仅通过冻结的视觉编码器处理图像，生成一次性文本输出并丢弃内部表征。我们提出DoYouRemember，一种三阶段架构，将重构性记忆引入MLLM：(1) 使用VQ-VAE将图像压缩为离散视觉令牌；(2) 通过LoRA微调的LLM联合关注视觉与文本令牌；(3) 利用扩散解码器从LLM隐藏状态中重构图像。在1,000个3D面部皮肤纹理图和99,000张未标注人脸图像上测试发现，LLM隐藏状态中约无可恢复的视觉信息——同一解码器在预LLM阶段（即VQ令牌）能生成清晰重建，而在后LLM阶段则输出纯噪声，证明模型理解图像但不真正记忆。在反向传播下训练共享记忆矩阵M时，因梯度衰减（O(1/sqrt(N))）导致系统失败。我们识别出三个根本原因，并证明局部EMA更新可全部解决：每张图像仅更新64个槽中的前8个，保持槽间多样性。最终获得的M（229K参数，压缩16倍）在未见测试图像上逼近VQ上限。扩展至1,024个槽时性能超越之（LPIPS 0.056 vs. 0.071），因连续表示避免了量化误差。我们以信息论框架统一这些发现：记忆是损毁压缩，回忆是解压缩，幻觉是损毁解压缩的固有属性而非缺陷。
原文摘要 · Abstract (English)
Human memory is reconstructive, not a faithful recording. Current multimodal LLMs (MLLMs) lack this capability: they process images through a frozen visual encoder, produce a one-shot text output, and discard internal representations. We present DoYouRemember, a three-stage architecture introducing reconstructive memory into MLLMs: (1) a VQ-VAE compresses images into discrete visual tokens, (2) a LoRA-fine-tuned LLM jointly attends to visual and text tokens, and (3) a Diffusion Decoder reconstructs images from the LLM's hidden states. On 1,000 3D facial skin texture maps and 99,000 unlabeled facial images, we find that LLM hidden states contain approximately zero recoverable visual information -- the same Decoder producing clear reconstructions from VQ-VAE tokens (pre-LLM) produces pure noise from LLM hidden states (post-LLM), demonstrating that the LLM understands images but does not remember them. Training a shared memory matrix M under backpropagation systematically fails due to gradient cancellation (O(1/sqrt(N)) attenuation). We identify three root causes and show that local EMA updating resolves all three: each image updates only its top-8 slots out of 64, preserving inter-slot diversity. The resulting M (229K parameters, 16x compressed) approaches the VQ upper bound on unseen test images. Scaling to 1,024 slots surpasses it (LPIPS 0.056 vs. 0.071), as M's continuous representation avoids VQ quantization error. We unify these findings under an information-theoretic framework: memory is lossy compression, recall is decompression, and hallucination is an inherent property of lossy decompression rather than a defect.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。