为长文档多模态问答设计结构化标注,提升重排序器对图文证据的感知能力。
What the Reranker Sees: Multi-Aspect Page Annotation for Long-Document Multimodal Question Answering

- 将候选文档转化为含图文描述、路径、实体标签等的语义记录,支持统一重排序。
- 在两个数据集上,重排序后F1提升至最高0.536,显著超越基线方法。
- 适合需要精准理解图表、表格和布局信息的复杂长文档问答场景。
长文档多模态问答通常采用检索-阅读流水线。在本研究中,瓶颈从检索召回转向重排序阶段的证据选择:在MMLongBench-Doc上,BGE-M3达到Recall@20 = 0.86,但F1@5仅为0.254;即使使用视觉检索器ColPali,F1@5也仅达0.332;仅依赖文本的重排序大模型即使上游已编码图像,仍会遗漏表格、图表与版式信息。为此提出Trident框架,包含两个互补模块:Trident-R为与检索器无关的LLM重排序器,将每个候选转换为包含视觉描述、章节路径、实体标签、多轴概念命中及文本片段的可读语义记录,并执行一次自适应K重排序;Trident-S为生成侧模块,在主题、实体与结构视角下提示VLM后再合成。在两个长文档数据集上,结合标注与重排序协议显著提升检索F1,在五个异构检索池中均超过最强自适应K基线PageIndex。不加标注的纯LLM重排序几乎不改变首项排名,表明性能提升源于结构化标注。Trident-S专为开放生成问题设计,使生成准确率最高提升6.6点。最佳配置在评估中成为最强下游问答流水线,两名LLM评委评分一致性kappa=0.913。
原文摘要 · Abstract (English)
Long-document visual question answering (VQA) over documents of tens to hundreds of pages mixing text, tables, charts, and figures typically follows retrieve-then-read pipelines. In our setting, the bottleneck shifts from retrieval recall to reranker-side evidence selection: on MMLongBench-Doc, BGE-M3 reaches Recall@20 = 0.86 but only F1@5 = 0.254, and even the visual retriever ColPali reaches only F1@5 = 0.332; a text-only rerank LLM seeing only raw snippets misses table, chart, and layout evidence even when the upstream retriever encoded images. We propose Trident, with two complementary components: Trident-R, a retriever-agnostic LLM reranker that converts each candidate into an LLM-readable semantic record, including a visual caption, section path, entity tags, multi-axis concept hits, and a text snippet, then performs a single adaptive-K rerank call; and Trident-S, a generation-side module that prompts the VLM under topical, entity, and structural lenses before synthesis. On two long-document datasets, the annotation+rerank protocol substantially improves retrieval F1 across five heterogeneous pools, with every reranked pool exceeding the strongest adaptive-K baseline PageIndex. An LLM rerank without the annotation barely changes first-hit ranking, indicating the lift comes from the structured annotation. Trident-S targets open-ended synthesis questions by design, adding up to 6.6 points in generation accuracy on these questions. The best Trident configuration is the strongest downstream QA pipeline in our evaluation, with rankings consistent across two LLM judges (kappa = 0.913).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。