70亿参数模型超越大模型，专精文档信息提取。
Extract-0: A Specialized Language Model for Document Information Extraction
- 用合成数据+低秩微调+强化学习优化，仅改0.53%参数。
- 在1000项任务上平均得分0.573，超GPT-4.1等大模型。
- 适合需要高效精准文档解析的场景，如金融、医疗。
本文提出Extract-0，一个70亿参数的专用语言模型，专为文档信息提取优化，性能超过参数量大几个数量级的模型。通过合成数据生成、基于低秩适配（LoRA）的监督微调，以及基于组相对策略优化（GRPO）的强化学习，Extract-0在1000个多样化文档抽取任务的基准上取得0.573的平均奖励，优于GPT-4.1（0.457）、o3（0.464）和GPT-4.1-2025（0.459）。训练采用记忆保持的合成数据生成管道，从多样文档源生成280,128个训练样本，并仅修改4040万（占总参数76.6亿的0.53%）模型权重实现参数高效微调。强化学习阶段引入基于语义相似性的新奖励函数，有效应对信息抽取中的固有歧义。研究证明，任务特化优化可使模型超越通用系统，同时大幅降低计算资源需求。
原文摘要 · Abstract (English)
This paper presents Extract-0, a 7-billion parameter language model specifically optimized for document information extraction that achieves performance exceeding models with parameter counts several orders of magnitude larger. Through a novel combination of synthetic data generation, supervised fine-tuning with Low-Rank Adaptation (LoRA), and reinforcement learning via Group Relative Policy Optimization (GRPO), Extract-0 achieves a mean reward of 0.573 on a benchmark of 1,000 diverse document extraction tasks, outperforming GPT-4.1 (0.457), o3 (0.464), and GPT-4.1-2025 (0.459). The training methodology employs a memory-preserving synthetic data generation pipeline that produces 280,128 training examples from diverse document sources, followed by parameterefficient fine-tuning that modifies only 0.53% of model weights (40.4M out of 7.66B parameters). The reinforcement learning phase introduces a novel semantic similarity-based reward function that handles the inherent ambiguity in information extraction tasks. This research demonstrates that task-specific optimization can yield models that surpass general-purpose systems while requiring substantially fewer computational resource.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。