轻量化多模态模型，用1.58位编码+记忆模块实现边缘设备高效图文生成
BitMar: Low-Bit Multimodal Fusion with Episodic Memory for Edge Devices
- 采用1.58位文本与视觉编码器，结合外部记忆库进行跨模态融合
- 在边缘设备上实现低延迟、小模型体积的高质量图文生成与理解
- 适合资源受限场景下的实时多模态应用，如智能摄像头、可穿戴设备
跨注意力变换器等多模态视觉-语言模型在语义定位和内容生成方面表现优异；然而其庞大的全精度主干网络难以部署于边缘设备。带有记忆增强的架构能更好利用历史上下文，但现有工作很少将其与激进的边缘量化结合。我们提出BitMar，一种量化多模态变换器，引入类人类的外置情景记忆，以在资源受限硬件上实现高效的图像-文本生成。BitMar采用1.58位编码器（文本为BitNet风格，视觉基于DiNOv2）生成紧凑嵌入，融合后用于查询固定大小的键值记忆库。向量检索时，BitNet解码器通过逐层条件化提升生成内容的上下文相关性。解码器还采用带滑动窗口机制的注意力消融模块，可在极低内存预算下处理长序列或流式输入。逐层条件化与滑动窗口注意力的结合实现了优良的质量-速度权衡，在低延迟下达成有竞争力的图像描述与多模态理解性能，模型体积小，非常适合边缘部署。
原文摘要 · Abstract (English)
Cross-attention transformers and other multimodal vision-language models excel at grounding and generation; however, their extensive, full-precision backbones make it challenging to deploy them on edge devices. Memory-augmented architectures enhance the utilization of past context; however, most works rarely pair them with aggressive edge-oriented quantization. We introduce BitMar, a quantized multimodal transformer that proposes an external human-like episodic memory for effective image-text generation on hardware with limited resources. BitMar utilizes 1.58-bit encoders, one for text (BitNet-style) and one for vision (DiNOv2-based), to create compact embeddings that are combined and used to query a fixed-size key-value episodic memory. During vector retrieval, the BitNet decoder applies per-layer conditioning, which increases the contextual relevance of generated content. The decoder also employs attention sinks with a sliding-window mechanism to process long or streaming inputs under tight memory budgets. The combination of per-layer conditioning and sliding-window attention achieves a strong quality-speed trade-off, delivering competitive captioning and multimodal understanding at low latency with a small model footprint. These characteristics make BitMar well-suited for edge deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。