无需人工规则，自动从日志中提取可读模板
SchemaCoder: Automatic Log Schema Extraction Coder with Residual Q-Tree Boosting
- 用残差问答树迭代优化，让大模型自动生成日志结构
- 在LogHub-2.0上比现有方法平均提升21.3%准确率
- 适合需要批量处理日志的运维与安全分析人员
日志模式提取是从海量日志数据中生成可读模板的过程，对系统监控和故障诊断至关重要，但传统方法高度依赖人工规则，效率低下。现有基于大语言模型（LLM）的方法仍需预设正则表达式，限制了自动化程度。为此，我们提出SchemaCoder，首个无需人工干预即可适配多种日志格式的全自动提取框架。其核心是创新的残差问答树（Q-Tree）增强机制，通过上下文分割将日志切分为语义块，利用嵌入采样选择代表性模式，并以分层问答驱动的LLM查询生成模板，再经文本残差演化优化器和残差提升持续迭代优化。在广泛使用的LogHub-2.0基准上，SchemaCoder相比最先进方法平均提升21.3%。
原文摘要 · Abstract (English)
Log schema extraction is the process of deriving human-readable templates from massive volumes of log data, which is essential yet notoriously labor-intensive. Recent studies have attempted to streamline this task by leveraging Large Language Models (LLMs) for automated schema extraction. However, existing methods invariably rely on predefined regular expressions, necessitating human domain expertise and severely limiting productivity gains. To fundamentally address this limitation, we introduce SchemaCoder, the first fully automated schema extraction framework applicable to a wide range of log file formats without requiring human customization within the flow. At its core, SchemaCoder features a novel Residual Question-Tree (Q-Tree) Boosting mechanism that iteratively refines schema extraction through targeted, adaptive queries driven by LLMs. Particularly, our method partitions logs into semantic chunks via context-bounded segmentation, selects representative patterns using embedding-based sampling, and generates schema code through hierarchical Q-Tree-driven LLM queries, iteratively refined by our textual-residual evolutionary optimizer and residual boosting. Experimental validation demonstrates SchemaCoder's superiority on the widely-used LogHub-2.0 benchmark, achieving an average improvement of 21.3% over state-of-the-arts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。