用大模型生成文档知识，提升小模型在陌生场景下的理解能力
DocKD: Knowledge Distillation from LLMs for Open-World Document Understanding Models
- 通过引入文档结构、关键词等外部信息增强大模型生成能力
- 生成数据训练的小模型在跨领域任务上超越人工标注数据训练的模型
- 特别适合需要泛化能力的开放世界文档理解场景
视觉文档理解（VDU）是一项挑战性任务，需处理多模态（文本与图像）及多样化布局（表单、表格等）。本文旨在通过从大语言模型（LLM）中蒸馏知识，提升小型VDU模型的泛化能力。研究发现，直接提示LLM常无法生成有效信息。为此，我们提出新框架DocKD，通过整合外部文档知识丰富数据生成过程：向LLM提供关键值对、布局信息和描述等文档元素，以激发开放式回答。实验表明，DocKD生成的文档标注质量高，显著优于不利用外部文档知识的直接知识蒸馏方法。此外，仅使用DocKD生成数据训练的学生模型，在域内任务上与人工标注数据训练模型相当，而在域外任务上则明显更优。
原文摘要 · Abstract (English)
Visual document understanding (VDU) is a challenging task that involves understanding documents across various modalities (text and image) and layouts (forms, tables, etc.). This study aims to enhance generalizability of small VDU models by distilling knowledge from LLMs. We identify that directly prompting LLMs often fails to generate informative and useful data. In response, we present a new framework (called DocKD) that enriches the data generation process by integrating external document knowledge. Specifically, we provide an LLM with various document elements like key-value pairs, layouts, and descriptions, to elicit open-ended answers. Our experiments show that DocKD produces high-quality document annotations and surpasses the direct knowledge distillation approach that does not leverage external document knowledge. Moreover, student VDU models trained with solely DocKD-generated data are not only comparable to those trained with human-annotated data on in-domain tasks but also significantly excel them on out-of-domain tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。