评测大模型生成云日志知识图谱的性能,发现少样本提示最有效。
Performance Evaluation of LLMs in Automated RDF Knowledge Graph Generation
- 用多模型少样本提示法自动提取日志中的实体关系三元组。
- Llama在少样本下达99.35%准确率,所有模型输出均为合法RDF。
- 提示设计比模型大小更重要,适合系统运维与安全分析者参考。
云系统产生大量异构日志数据,包含关键基础设施、应用和安全信息。将这些日志转化为RDF三元组可将其集成到知识图谱中,提升可解释性、根因分析与跨服务推理能力。大语言模型(LLMs)为自动化生成知识图谱提供了新路径,但其在复杂云日志上的效果尚不明确。本文构建了两个流水线的控制框架,系统处理半结构化日志数据。提取流水线整合多个LLM识别相关实体与关系,自动生成主语-谓语-宾语三元组;验证流水线采用语法与语义指标评估准确性、完整性和质量。由于缺乏公开真实数据集,我们基于OpenStack日志手动标注并结合本体方法构建了参考性的Log-to-KG数据集,实现客观基准。结果表明,少样本学习是最优策略,其中Llama取得99.35% F1分数且100%输出合法RDF;Qwen、NuExtract与Gemma在少样本提示下表现良好,链式思维方法也保持相近准确率。单样本提示提供轻量高效替代方案,而零样本及树状思维、自检、生成多个等进阶策略表现显著更差。研究强调上下文示例与提示设计对准确提取的关键作用,并揭示不同模型架构的局限性。
原文摘要 · Abstract (English)
Cloud systems generate large, heterogeneous log data containing critical infrastructure, application, and security information. Transforming these logs into RDF triples enables their integration into knowledge graphs, improving interpretability, root-cause analysis, and cross-service reasoning beyond what raw logs allow. Large Language Models (LLMs) offer a promising approach to automate RDF knowledge graph generation; however, their effectiveness on complex cloud logs remains largely unexplored. In this paper, we evaluate multiple LLM architectures and prompting strategies for automated RDF extraction using a controlled framework with two pipelines for systematically processing semi-structured log data. The extraction pipeline integrates multiple LLMs to identify relevant entities and relationships, automatically generating subject-predicate-object triples. These outputs are evaluated using a dedicated validation pipeline with both syntactic and semantic metrics to assess accuracy, completeness, and quality. Due to the lack of public ground-truth datasets, we created a reference Log-to-KG dataset from OpenStack logs using manual annotation and ontology-driven methods, enabling objective baseline. Our analysis shows that Few-Shot learning is the most effective strategy, with Llama achieving a 99.35% F1 score and 100% valid RDF output while Qwen, NuExtract, and Gemma also perform well under Few-Shot prompting, with Chain-of-Thought approaches maintaining similar accuracy. One-Shot prompting offers a lighter but effective alternative, while Zero-Shot and advanced strategies such as Tree-of-Thought, Self-Critique, and Generate-Multiple perform substantially worse. These results highlight the importance of contextual examples and prompt design for accurate RDF extraction and reveal model-specific limitations across LLM architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。