评测企业文档处理全流程，发现各环节表现不连贯且生成常遗漏信息。
Benchmarking Complex Multimodal Document Processing Pipelines: A Unified Evaluation Framework for Enterprise AI
- 构建统一框架，从解析到生成全程评估多模态文档流水线。
- 混合检索略胜于BM25（nDCG@5为0.92），但密集向量检索仅0.83。
- 生成答案准确但常缺内容（完整度仅0.40），适合关注系统真实表现者。
当前企业文档AI多为流水线结构：解析、索引、检索、生成。尽管各阶段单独研究充分，但整体系统评估仍具挑战。为此，我们构建了EnterpriseDocBench：在六类企业领域的真实文档上，统一评估解析精度、索引效率、检索相关性与生成事实一致性。使用相同GPT-5生成器，测试了BM25、密集嵌入和混合三种管道。结果显示，混合检索略胜一筹（nDCG@5=0.92），优于密集嵌入（0.83）；长文档与短文档的幻觉率分别为28.1%和23.8%，中等长度最低（9.2%）。跨阶段相关性极弱（如解析→生成仅r=0.17），表明质量并非级联传递。系统在陈述事实上的准确率达85.5%，但答案完整度平均仅0.40，说明回答正确却常遗漏关键信息。该差距对实际部署影响更大。文中还提出三种参考架构（ColPali、ColQwen2、基于代理复杂度的路由），尚未端到端集成。框架、指标、基线与数据采集脚本将在录用后开源。
原文摘要 · Abstract (English)
Most enterprise document AI today is a pipeline. Parse, index, retrieve, generate. Each of those stages has been studied to death on its own -- what's still hard is evaluating the system as a whole. We built EnterpriseDocBench to take a swing at it: parsing fidelity, indexing efficiency, retrieval relevance, and generation groundedness, all on the same corpus. The corpus is built from public, permissively licensed documents across six enterprise domains (five represented in the current pilot). We ran three pipelines through it -- BM25, dense embedding, and a hybrid -- all with the same GPT-5 generator. The headline numbers: hybrid retrieval narrowly beats BM25 (nDCG@5 of 0.92 vs. 0.91), and both beat dense embedding (0.83). Hallucination doesn't grow monotonically with document length -- short documents and very long ones both hallucinate more than medium ones (28.1% and 23.8% vs. 9.2%). Cross-stage correlations are very weak: parsing->retrieval r=0.14, parsing->generation r=0.17, retrieval->generation 0.02. If quality were cascading the way most of us assume, those numbers would be much higher; they aren't. Design caveats are real (parsing fixed, generator shared, automated proxy metrics) and we don't oversell the result. One result that genuinely surprised us: factual accuracy on stated claims is 85.5%, but answer completeness averages 0.40. The system is right when it answers -- it just leaves things out. That gap matters more for real deployments than the headline accuracy number does. We also describe three reference architectures (ColPali, ColQwen2, agentic complexity-based routing) which are not yet integrated end-to-end. Framework, metrics, baselines, and collection scripts will be released open-source on acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。