用推理强化学习提升大模型日志分析能力,减少幻觉并增强泛化。
R-Log: Incentivizing Log Analysis Capability in LLMs via Reasoning-based Reinforcement Learning
- 通过模拟工程师的分步推理过程建立日志分析能力
- 在真实日志上跨任务表现优于现有方法,未见场景提升228.05%
- 兼顾效率与效果,快速版速度提升5倍且保持93%性能
现代软件系统中日志数据日益复杂,促使使用大语言模型(LLMs)进行自动化日志分析。当前方法通常基于日志-标签对进行直接监督微调(SFT),但加剧了通用模型与专业日志间的领域差异,导致过拟合。此外,SFT的损失计算不均衡常使长上下文淹没关键细节,引发幻觉。为此,我们提出R-Log,一种基于推理的新型范式,模仿工程师结构化的分步分析流程,通过学习结论背后的规律提升泛化能力。我们进一步采用强化学习(RL)在模拟运维环境中优化模型,直接奖励正确结果以减少幻觉。R-Log首先在包含2000+条推理轨迹的精选数据集上冷启动,受13种人工运维策略引导,建立初始推理能力;随后通过联合奖励函数进行强化学习精炼。实证评估显示,R-Log在真实日志上五项分析任务中均优于现有方法,尤其在未见场景下提升达228.05%。我们还设计了R-Log-fast,实现5倍提速同时保留93%效能。
原文摘要 · Abstract (English)
The growing complexity of log data in modern software systems has prompted the use of Large Language Models (LLMs) for automated log analysis. Current approaches typically rely on direct supervised fine-tuning (SFT) on log-label pairs. However, this exacerbates the domain discrepancy between general-purpose LLMs and specialized log data, causing overfitting. Furthermore, SFT's imbalanced loss computation often allows lengthy contexts to overwhelm critical, concise details in model answers, leading to hallucinations. To address these limitations, we propose R-Log, a novel reasoning-based paradigm that mirrors the structured, step-by-step analytical process of human engineers. This approach enhances generalizability by learning the underlying rules behind conclusions. We further employ Reinforcement Learning (RL) to optimize the model within a simulated O&M environment, thereby reducing hallucinations by directly rewarding correct outcomes. R-Log is first cold-started on a curated dataset of 2k+ reasoning trajectories, guided by 13 strategies from manual O&M practices, to establish an initial reasoning capability. This ability is then refined via RL using a joint reward function. Empirical evaluations on real-world logs show that R-Log outperforms existing methods across five log analysis tasks, particularly in unseen scenarios (by 228.05%). We also designed R-Log-fast with 5x speedup while keeping 93% of the efficacy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。