让手机在低资源下高效回忆多模态信息
Recall: Empowering Multimodal Embedding for Edge Devices
- 粗粒度嵌入+查询过滤，提升检索效率
- 高吞吐、高精度，内存能耗极低
- 适合移动设备上的实时记忆应用
人类记忆具有天然遗忘特性。为应对这一问题，多模态嵌入模型被提出，将现实世界中的多样化数据映射到统一的嵌入空间，可高效检索，帮助移动用户回忆过往信息。然而，随着模型复杂度上升，资源消耗增加，导致吞吐量下降、计算负担加重，限制了在移动设备上的部署。本文提出RECALL，一种专为资源受限移动环境优化的本地化多模态嵌入系统。该系统通过生成粗粒度嵌入并结合基于查询的过滤机制，实现高吞吐、高精度的检索。实验表明，RECALL在保持高质量嵌入的同时，展现出优异的吞吐性能，且运行时对内存和能耗几乎无影响。
原文摘要 · Abstract (English)
Human memory is inherently prone to forgetting. To address this, multimodal embedding models have been introduced, which transform diverse real-world data into a unified embedding space. These embeddings can be retrieved efficiently, aiding mobile users in recalling past information. However, as model complexity grows, so do its resource demands, leading to reduced throughput and heavy computational requirements that limit mobile device implementation. In this paper, we introduce RECALL, a novel on-device multimodal embedding system optimized for resource-limited mobile environments. RECALL achieves high-throughput, accurate retrieval by generating coarse-grained embeddings and leveraging query-based filtering for refined retrieval. Experimental results demonstrate that RECALL delivers high-quality embeddings with superior throughput, all while operating unobtrusively with minimal memory and energy consumption.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。