用渐进式元提示提升小模型日志解析精度与速度
MicLog: Towards Accurate and Efficient LLM-based Log Parsing via Progressive Meta In-Context Learning
- 通过动态选例和加权聚类增强小模型的上下文学习能力
- 在Loghub-2.0上准确率高出10.3%,解析时间缩短42.4%
- 适合需要高效低耗日志分析的系统运维与安全团队
日志解析将半结构化日志转化为结构化模板,是下游分析的基础。传统语法语义解析器在日志语义变化和数据稀缺时表现不佳。近年基于大模型的解析器虽利用上下文学习提取语义,但存在两问题:一是上下文学习能力未被充分挖掘,尤其在动态示例选择和跨域泛化上表现不一;二是调用大模型耗时且昂贵。为此,本文提出首个渐进式元上下文学习(ProgMeta-ICL)日志解析框架MicLog,结合元学习与小规模开源大模型(Qwen-2.5-3B)的上下文学习。具体而言,MicLog:i)通过零样本到k样本的渐进式元上下文学习范式,采用加权DBSCAN候选采样与增强BM25示例选择,提升上下文学习能力;ii)引入多级预查询缓存,动态匹配并优化近期解析的模板以加速处理。在Loghub-2.0上的实验表明,MicLog相比当前最优解析器准确率提升10.3%,解析时间减少42.4%。
原文摘要 · Abstract (English)
Log parsing converts semi-structured logs into structured templates, forming a critical foundation for downstream analysis. Traditional syntax and semantic-based parsers often struggle with semantic variations in evolving logs and data scarcity stemming from their limited domain coverage. Recent large language model (LLM)-based parsers leverage in-context learning (ICL) to extract semantics from examples, demonstrating superior accuracy. However, LLM-based parsers face two main challenges: 1) underutilization of ICL capabilities, particularly in dynamic example selection and cross-domain generalization, leading to inconsistent performance; 2) time-consuming and costly LLM querying. To address these challenges, we present MicLog, the first progressive meta in-context learning (ProgMeta-ICL) log parsing framework that combines meta-learning with ICL on small open-source LLMs (i.e., Qwen-2.5-3B). Specifically, MicLog: i) enhances LLMs' ICL capability through a zero-shot to k-shot ProgMeta-ICL paradigm, employing weighted DBSCAN candidate sampling and enhanced BM25 demonstration selection; ii) accelerates parsing via a multi-level pre-query cache that dynamically matches and refines recently parsed templates. Evaluated on Loghub-2.0, MicLog achieves 10.3% higher parsing accuracy than the state-of-the-art parser while reducing parsing time by 42.4%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。