用大模型结构先验提升少样本文档布局分析效果
LLM-Guided Probabilistic Fusion for Label-Efficient Document Layout Analysis
- 融合视觉预测与大模型结构先验，通过概率加权生成更准伪标签
- 仅用5%标签即达88.2 AP，优于标准半监督方法0.6点以上
- 支持私有部署，开源模型损失小，适合资源有限场景
尽管半监督学习取得进展，文档布局理解仍依赖大量数据。本文提出一种框架，通过合理概率加权，将文本预训练大模型的结构先验与视觉预测融合，生成高质量伪标签。在无标签文档上，使用OCR-LLM管道推断层级区域，并与教师检测器输出通过逆方差融合结合。轻量级SwiftFormer（26M参数）在仅5%标签下达到88.2±0.3 AP（PubLayNet）。应用于文档预训练的LayoutLMv3（133M参数）时，达89.7±0.4 AP，显著优于标准半监督版（89.1±0.4，p=0.02），接近需超百万页多模态预训练的UDOP（89.8 AP）。关键发现：(1) 学习的实例自适应门控比固定权重提升0.9 AP，且基于数据依赖的PAC界正确预测收敛；(2) 开源大模型支持隐私保护部署，损失小（Llama-3-70B：轻量级87.1，LayoutLMv3为89.4）；(3) 大模型实现精准语义消歧（18.7%案例，增益+3.8 AP），超越简单文本启发式。系统成本：每5万页约12美元GPT-4o-mini API费用或17 GPU小时本地部署。
原文摘要 · Abstract (English)
Document layout understanding remains data-intensive despite advances in semi-supervised learning. We present a framework that enhances semi-supervised detection by fusing visual predictions with structural priors from text-pretrained LLMs via principled probabilistic weighting. Given unlabeled documents, an OCR-LLM pipeline infers hierarchical regions which are combined with teacher detector outputs through inverse-variance fusion to generate refined pseudo-labels.Our method demonstrates consistent gains across model scales. With a lightweight SwiftFormer backbone (26M params), we achieve 88.2$\pm$0.3 AP using only 5\% labels on PubLayNet. When applied to document-pretrained LayoutLMv3 (133M params), our fusion framework reaches 89.7$\pm$0.4 AP, surpassing both LayoutLMv3 with standard semi-supervised learning (89.1$\pm$0.4 AP, p=0.02) and matching UDOP~\cite{udop} (89.8 AP) which requires 100M+ pages of multimodal pretraining. This demonstrates that LLM structural priors are complementary to both lightweight and pretrained architectures. Key findings include: (1) learned instance-adaptive gating improves over fixed weights by +0.9 AP with data-dependent PAC bounds correctly predicting convergence; (2) open-source LLMs enable privacy-preserving deployment with minimal loss (Llama-3-70B: 87.1 AP lightweight, 89.4 AP with LayoutLMv3); (3) LLMs provide targeted semantic disambiguation (18.7\% of cases, +3.8 AP gain) beyond simple text heuristics.Total system cost includes \$12 for GPT-4o-mini API or 17 GPU-hours for local Llama-3-70B per 50K pages, amortized across training runs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。