用日志严重性分类测试小模型理解能力,发现检索增强能大幅提升性能。
Benchmarking Small Language Models and Small Reasoning Language Models on System Log Severity Classification
- 用零样本、少样本和检索增强提示评估9个小型语言模型
- Qwen3-4B在检索增强下准确率达95.64%,0.6B版本也达88.12%
- 模型推理速度差异大,部分模型虽慢但准确率不足10%
系统日志对现代计算基础设施的监控与诊断至关重要,但其规模与复杂性需要可靠的自动化解读。由于严重性等级是日志消息中的预定义元数据,单纯分类任务本身实用价值有限,难以反映模型对日志的真实理解能力。本文主张将严重性分类作为探测运行时日志理解力的基准,而非终点任务。基于来自真实生产环境Linux服务器的journalctl数据,我们评估了9个小型语言模型(SLMs)和小型推理语言模型(SRLMs),采用零样本、少样本及检索增强生成(RAG)提示策略。结果显示显著分层:使用RAG时,Qwen3-4B达到最高准确率95.64%;Gemma3-1B从少样本下的20.25%提升至RAG下的85.28%;即使微小的Qwen3-0.6B在检索增强下也达88.12%准确率。相反,部分SRLMs如Qwen3-1.7B和DeepSeek-R1-Distill-Qwen-1.5B在搭配RAG后性能大幅下降。效率测量进一步区分模型:多数Gemma与Llama变体每条日志推理时间低于1.2秒,而Phi-4-Mini-Reasoning每条耗时超228秒且准确率低于10%。结果表明,模型架构设计、训练目标与在严格输出约束下整合检索信息的能力共同决定性能。该基准强调可部署的小模型,契合数字孪生(DT)系统的实时需求,显示严重性分类可作为评估模型能力与实时部署潜力的视角,对根因分析(RCA)和广泛DT集成具有意义。
原文摘要 · Abstract (English)
System logs are crucial for monitoring and diagnosing modern computing infrastructure, but their scale and complexity require reliable and efficient automated interpretation. Since severity levels are predefined metadata in system log messages, having a model merely classify them offers limited standalone practical value, revealing little about its underlying ability to interpret system logs. We argue that severity classification is more informative when treated as a benchmark for probing runtime log comprehension rather than as an end task. Using real-world journalctl data from Linux production servers, we evaluate nine small language models (SLMs) and small reasoning language models (SRLMs) under zero-shot, few-shot, and retrieval-augmented generation (RAG) prompting. The results reveal strong stratification. Qwen3-4B achieves the highest accuracy at 95.64% with RAG, while Gemma3-1B improves from 20.25% under few-shot prompting to 85.28% with RAG. Notably, the tiny Qwen3-0.6B reaches 88.12% accuracy despite weak performance without retrieval. In contrast, several SRLMs, including Qwen3-1.7B and DeepSeek-R1-Distill-Qwen-1.5B, degrade substantially when paired with RAG. Efficiency measurements further separate models: most Gemma and Llama variants complete inference in under 1.2 seconds per log, whereas Phi-4-Mini-Reasoning exceeds 228 seconds per log while achieving <10% accuracy. These findings suggest that (1) architectural design, (2) training objectives, and (3) the ability to integrate retrieved context under strict output constraints jointly determine performance. By emphasizing small, deployable models, this benchmark aligns with real-time requirements of digital twin (DT) systems and shows that severity classification serves as a lens for evaluating model competence and real-time deployability, with implications for root cause analysis (RCA) and broader DT integration.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。