用微调视觉语言模型,精准将马来西亚财报表格转为Markdown。
Fine-Tuning Vision-Language Models for Markdown Conversion of Financial Tables in Malaysian Audited Financial Reports
- 基于Qwen2.5-VL-7B微调,结合数据增强与LoRA策略。
- 在100张测试表上达到92.20%准确率和96.53%结构相似度。
- 性能超越大模型与商业工具,推理更快更高效。
从财务文档中准确提取并表示表格结构仍是文档理解中的关键挑战,尤其在监管与分析场景中。本研究针对马来西亚审计财报中表格转为Markdown格式的难题,该任务受旋转布局、多级标题及隐含结构线索影响。我们提出一种基于Qwen2.5-VL-7B的微调视觉语言模型(VLM),专用于从文档图像生成高保真Markdown。方法包含2,152张图像-文本对的标注数据集及采用LoRA的监督微调策略。评估使用双框架:基于规则的LLM-as-a-judge进行细粒度准确性评估,以及新提出的基于树编辑距离的Markdown TEDS指标衡量整体结构保真度。模型在标准测试中达92.20%总体准确率与96.53%的TEDS得分,显著优于其基线模型、更大规模VLM及具备推理能力的专用模型。相比自托管方案,推理时间大幅降低。性能还超过广泛使用的OpenAI GPT-4o与Gemini 2.5 Flash等商用模型。结果表明,领域特定微调是连接非结构化财务文档与下游自动化之间的有效且高效路径,无需大型模型的计算开销即可媲美其表现。
原文摘要 · Abstract (English)
Accurately extracting and representing the structure of tabular data from financial documents remains a critical challenge in document understanding, particularly for regulatory and analytical use cases. This study addresses the complexity of converting financial tables from Malaysian audited financial reports into Markdown format, a task complicated by rotated layouts, multi-level headers, and implicit structural cues. We propose a fine-tuned vision-language model (VLM), based on Qwen2.5-VL-7B, optimized for high-fidelity Markdown generation from document images. Our approach includes a curated dataset of 2,152 image-text pairs with augmentations and a supervised fine-tuning strategy using LoRA. To assess performance, we evaluated our model on 100 out-of-sample tables using a dual framework: a criteria-based LLM-as-a-judge for fine-grained accuracy and our novel Markdown Tree-Edit-Distance-based Similarity (TEDS) metric for holistic structural fidelity. Our model achieves a 92.20% overall accuracy on the criteria-based assessment and a 96.53% Markdown TEDS score. This performance significantly surpasses its Qwen2.5-VL-7B base model, larger-scale VLMs, and specialized reasoning-enabled models. Compared to these self-hosted alternatives, it also significantly reduces inference time. Furthermore, its accuracy exceeds that of widely used proprietary models such as OpenAI's GPT-4o and Gemini 2.5 Flash. These results demonstrate that domain-specific fine-tuning provides an effective and efficient method to bridge the gap between unstructured financial documents and downstream automation, rivalling much larger and more general models without their computational overhead.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。