用大模型让文档编辑更智能，一键改文字、图片和布局。
DocEdit-v2: Document Structure Editing Via Multimodal LLM Grounding
- 通过多模态理解用户指令，精准定位要修改的区域并转为可执行命令。
- 在真实文档数据集上，指令生成准确率提升2%-33%，区域检测提升12%-31%。
- 适合需要自动化文档修改的办公、设计人员，尤其支持复杂布局调整。
文档结构编辑涉及根据用户请求对文档图像中的文本、视觉内容和版式元素进行局部操作。以往工作表明，将用户请求与文档图像精准对齐，并准确识别结构组件及其属性仍是关键挑战。为此，我们提出DocEdit-v2，一个利用大型多模态模型（LMMs）实现端到端文档编辑的新框架。该框架包含三个创新组件：(1) Doc2Command，同步定位编辑兴趣区域（RoI）并消歧用户编辑请求为具体编辑命令；(2) 基于LLM的命令重表述提示，将原本针对专业软件设计的指令转化为适用于通用大模型的编辑指令；(3) 利用GPT-4V、Gemini等大型多模态模型处理输出，解析文档布局，在定位的区域上执行编辑并生成新文档图像。在DocEdit数据集上的大量实验表明，DocEdit-v2在编辑命令生成（提升2%-33%）、RoI边界框检测（提升12%-31%）以及整体文档编辑任务（提升1%-12%）上均显著优于强基线方法。
原文摘要 · Abstract (English)
Document structure editing involves manipulating localized textual, visual, and layout components in document images based on the user's requests. Past works have shown that multimodal grounding of user requests in the document image and identifying the accurate structural components and their associated attributes remain key challenges for this task. To address these, we introduce the DocEdit-v2, a novel framework that performs end-to-end document editing by leveraging Large Multimodal Models (LMMs). It consists of three novel components: (1) Doc2Command, which simultaneously localizes edit regions of interest (RoI) and disambiguates user edit requests into edit commands; (2) LLM-based Command Reformulation prompting to tailor edit commands originally intended for specialized software into edit instructions suitable for generalist LMMs. (3) Moreover, DocEdit-v2 processes these outputs via Large Multimodal Models like GPT-4V and Gemini, to parse the document layout, execute edits on grounded Region of Interest (RoI), and generate the edited document image. Extensive experiments on the DocEdit dataset show that DocEdit-v2 significantly outperforms strong baselines on edit command generation (2-33%), RoI bounding box detection (12-31%), and overall document editing (1-12\%) tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。