arXiv:2508.08589cs.CV2025-08ICCV被引 17

用强化学习让文档理解模型自己优化推理过程,更透明可信。

DocThinker: Explainable Multimodal Large Language Models with Rule-based Reinforcement Learning for Document Understanding

  • 通过规则奖励机制动态学习推理策略,替代固定思维链
  • 在多个数据集上提升泛化能力,且推理步骤更易懂
  • 适合法律、医疗等对可解释性要求高的场景

多模态大语言模型在文档理解任务中表现卓越,但其推理过程缺乏透明性,难以在法律、金融、医疗等高风险领域建立信任。现有方法依赖监督微调的固定思维链,存在灾难性遗忘、适应性差和跨领域泛化能力弱的问题。本文提出DocThinker,一种基于规则的强化学习框架,实现推理时的动态优化。该模型不依赖静态思维链模板,而是通过策略学习自主调整推理路径,生成结构化推理过程、重述问题、支持答案的感兴趣区域(RoI)及最终答案。结合多目标规则奖励与KL约束优化,有效缓解灾难性遗忘,提升适应性和可解释性。在多个基准测试中,实验表明该方法显著增强泛化性能,并产生更符合人类理解的推理步骤。结果表明,强化学习是提升多模态模型可解释性与适应性的有力途径。代码将开源于https://github.com/wenwenyu/DocThinker。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in document understanding. However, their reasoning processes remain largely black-box, making it difficult to ensure reliability and trustworthiness, especially in high-stakes domains such as legal, financial, and medical document analysis. Existing methods use fixed Chain-of-Thought (CoT) reasoning with supervised fine-tuning (SFT) but suffer from catastrophic forgetting, poor adaptability, and limited generalization across domain tasks. In this paper, we propose DocThinker, a rule-based Reinforcement Learning (RL) framework for dynamic inference-time reasoning. Instead of relying on static CoT templates, DocThinker autonomously refines reasoning strategies via policy learning, generating explainable intermediate results, including structured reasoning processes, rephrased questions, regions of interest (RoI) supporting the answer, and the final answer. By integrating multi-objective rule-based rewards and KL-constrained optimization, our method mitigates catastrophic forgetting and enhances both adaptability and transparency. Extensive experiments on multiple benchmarks demonstrate that DocThinker significantly improves generalization while producing more explainable and human-understandable reasoning steps. Our findings highlight RL as a powerful alternative for enhancing explainability and adaptability in MLLM-based document understanding. Code will be available at https://github.com/wenwenyu/DocThinker.

文档理解可解释性强化学习多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。