轻量级多模态文档理解模型，可在边缘设备高效运行
DocSLM: A Small Vision-Language Model for Long Multimodal Document Understanding
- 分层压缩编码图文版面信息，减少82%视觉令牌
- 通过流式弃权机制实现长文档渐进处理，降低71%延迟
- 适合资源受限的边缘设备部署，性能优于主流方法
大型视觉语言模型在长而复杂的文档上展现出强大的多模态推理能力，但其高内存开销使其难以在资源受限的边缘设备上部署。本文提出DocSLM，一种专为受限内存环境设计的小型视觉语言模型，用于长文档理解。该模型引入分层多模态压缩器，将每页的视觉、文本和版面信息联合编码为固定长度序列，显著降低内存占用，同时保留局部与全局语义。为支持任意长度输入的可扩展处理，我们设计了流式弃权机制，通过熵基不确定性校准器对文档片段进行顺序处理，并过滤低置信度响应。在多个长多模态文档基准测试中，DocSLM在仅使用82%视觉令牌、75%参数量和71%延迟的情况下，达到或超过现有最优方法性能，实现在轻量级边缘设备上的可靠多模态文档理解。代码与模型已开源：https://github.com/Tanveer81/DocSLM.git。
原文摘要 · Abstract (English)
Large Vision-Language Models (LVLMs) have demonstrated strong multimodal reasoning capabilities on long and complex documents. However, their high memory footprint makes them impractical for deployment on resource-constrained edge devices. We present DocSLM, an efficient Small Vision-Language Model designed for long-document understanding under constrained memory resources. DocSLM incorporates a Hierarchical Multimodal Compressor that jointly encodes visual, textual, and layout information from each page into a fixed-length sequence, greatly reducing memory consumption while preserving both local and global semantics. To enable scalable processing over arbitrarily long inputs, we introduce a Streaming Abstention mechanism that operates on document segments sequentially and filters low-confidence responses using an entropy-based uncertainty calibrator. Across multiple long multimodal document benchmarks, DocSLM matches or surpasses state-of-the-art methods while using 82\% fewer visual tokens, 75\% fewer parameters, and 71\% lower latency, delivering reliable multimodal document understanding on lightweight edge devices. Code and Model are available in https://github.com/Tanveer81/DocSLM.git.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。