用真实业务文档构建规则流程数据集,提升大模型对复杂逻辑的解析能力。
Business as Rulesual: A Benchmark and Framework for Business Rule Flow Modeling with LLMs
- 设计结构化推理框架ExIde,通过伪代码生成实现规则显式建模。
- 在409份真实文档上验证,执行性引导比普通提示提升显著效果。
- 适合需要自动化处理复杂规章的企业客户和系统开发者。
从非结构化业务文档中提取结构化流程知识是流程自动化的关键瓶颈。现有研究多聚焦于线性操作流(如食谱),但对真实世界监管与行政文件中常见的条件分支、并行执行等复杂逻辑覆盖不足。同时,现有基准数据集受限于简单模式和浅层依赖关系,阻碍了逻辑感知大模型的发展。为此,我们提出BREX,一个精心构建的基准数据集,包含409个真实业务文档和2,855条专家标注规则,覆盖30余个垂直领域,涵盖科研、工业、行政及金融法规。我们进一步提出ExIde框架,探索五种不同提示策略,从隐式语义对齐到可执行伪代码生成,实现规则依赖的显式建模,并提供无需微调即可部署的开箱即用方案。我们在13个主流大模型上评估ExIde,结果表明,可执行接地作为更强归纳偏置,显著优于标准提示;且推理优化模型在追踪长程与非线性规则依赖方面表现更优。
原文摘要 · Abstract (English)
Extracting structured procedural knowledge from unstructured business documents is a critical yet unresolved bottleneck in process automation. While prior work has focused on extracting linear action flows from instructional texts, such as recipes, it has insufficiently addressed the complex logical structures, including conditional branching and parallel execution, that are pervasive in real-world regulatory and administrative documents. Furthermore, existing benchmarks are limited by simplistic schemas and shallow logical dependencies, restricting progress toward logic-aware large language models.To bridge this Logic Gap, we introduce BREX, a carefully curated benchmark comprising 409 real-world business documents and 2,855 expert-annotated rules. Unlike prior datasets centered on narrow service scenarios, BREX spans over 30 vertical domains, covering scientific, industrial, administrative, and financial regulations. We further propose ExIde, a structure-aware reasoning framework that investigates five distinct prompting strategies, ranging from implicit semantic alignment to executable grounding via pseudo-code generation. This enables explicit modeling of rule dependencies and provides an out-of-the-box framework for different business customers without finetuning their own large language models. We benchmark ExIde using 13 state-of-the-art large language models. Our extensive evaluation reveals that executable grounding serves as a superior inductive bias, significantly outperforming standard prompts in rule extraction. In addition, reasoning-optimized models demonstrate a distinct advantage in tracing long-range and non-linear rule dependencies compared to standard instruction-tuned models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。