用大模型自动提取历史事件,结合检索增强提升精度,还能转成可形式化验证的证明。
Reasoning with RAGged events: RAG-Enhanced Event Knowledge Base Construction and reasoning with proof-assistants
- 多模型结合RAG等策略自动抽取历史事件,避免人工构建成本
- RAG显著提升坐标和元数据准确性,但大模型表现差异极大
- 将事件转为Coq语言,实现因果链、时间计算等高级推理
从叙事文本中提取结构化历史事件表示仍需大量人工工作。尽管RDF/OWL推理器支持图推理,但仅限于一阶逻辑片段,难以深入进行时空与语义分析。本文通过多个大模型(GPT-4、Claude、Llama 3.2)结合纯基础生成、知识图谱增强及检索增强生成(RAG)三种策略,自动构建历史事件知识库,并在修昔底德文本上进行全面评估。结果表明:不同增强策略优化不同维度——基础生成在覆盖广度上最优(使用Claude与GPT-4),而RAG显著提升坐标精度与元数据完整性。模型架构决定增强敏感性:大模型基线性能强,微调提升稳定;而Llama 3.2表现波动剧烈,从优秀到完全失败。进一步开发自动化流水线,将提取出的RDF表示转换为Coq证明助手规格,实现高阶推理,包括多步因果验证、公元前日期的时间算术运算以及对历史因果关系的形式化证明。Coq形式化验证表明,RAG发现的事件类型是领域内合法的语义结构,非本体论违规。
原文摘要 · Abstract (English)
Extracting structured computational representations of historical events from narrative text remains computationally expensive when constructed manually. While RDF/OWL reasoners enable graph-based reasoning, they are limited to fragments of first-order logic, preventing deeper temporal and semantic analysis. This paper addresses both challenges by developing automatic historical event extraction models using multiple LLMs (GPT-4, Claude, Llama 3.2) with three enhancement strategies: pure base generation, knowledge graph enhancement, and Retrieval-Augmented Generation (RAG). We conducted comprehensive evaluations using historical texts from Thucydides. Our findings reveal that enhancement strategies optimize different performance dimensions rather than providing universal improvements. For coverage and historical breadth, base generation achieves optimal performance with Claude and GPT-4 extracting comprehensive events. However, for precision, RAG enhancement improves coordinate accuracy and metadata completeness. Model architecture fundamentally determines enhancement sensitivity: larger models demonstrate robust baseline performance with incremental RAG improvements, while Llama 3.2 shows extreme variance from competitive performance to complete failure. We then developed an automated translation pipeline converting extracted RDF representations into Coq proof assistant specifications, enabling higher-order reasoning beyond RDF capabilities including multi-step causal verification, temporal arithmetic with BC dates, and formal proofs about historical causation. The Coq formalization validates that RAG-discovered event types represent legitimate domain-specific semantic structures rather than ontological violations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。