用可解释性指导模型压缩，让文档问答模型更轻更快
Interpret, prune and distill Donut : towards lightweight VLMs for VQA on document
- 通过分析模型内部机制，识别关键组件进行保留与简化
- 压缩后模型在DocVQA上保持高精度，推理速度提升显著
- 适合需要高效部署的文档智能场景，如移动端或边缘设备
近期视觉文档理解依赖于大型多模态模型（如Donut），可在无需光学字符识别的情况下完成文档级视觉问答。然而这些模型计算开销大，难以用于实时或资源受限场景。本文通过知识蒸馏训练轻量级学生模型，并利用机制可解释性指导学生网络结构设计。通过分析内部计算过程，明确哪些子模块应保留、近似、跳过或重参数化。由此提出Donut-MINT（基于机制可解释性的网络剪枝），在保持DocVQA基准性能的同时，显著降低推理时间和内存占用。该方法将模型压缩重构为‘电路发现’过程，连接可解释性研究与实际多模态模型部署。
原文摘要 · Abstract (English)
Recent advances in Visually-rich Document Understanding rely on large Vision-Language Models like Donut, which perform document-level Visual Question Answering without Optical Character Recognition. Despite their effectiveness, these models are too costly for real-time or resource-constrained applications. We investigate model compression through knowledge distillation, training compact student models from a larger teacher. We leverage mechanistic interpretability to drive student architecture design within this framework. By analyzing internal computations, we identify essential subcomponents to retain, while having a clear view of which subcomponents should be approximated, skipped, or reparametrized based on their function. This approach yields Donut-MINT (Mechanistic Interpretability-based Network Trimming), a pruned Donut variant that reduces inference time and memory usage while maintaining strong performance on DocVQA, a standard benchmark for document Visual Question Answering. Our method reframes compression as circuit discovery, bridging interpretability research and practical Vision-Language Model deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。