用大模型检测日志安全问题,发现能识别但难修复。
Towards Secure Logging: Characterizing and Benchmarking Logging Code Security Issues with LLMs
- 构建日志安全问题分类体系与真实案例数据集
- 大模型检测准确率仅12.9%~52.5%,修复成功率低
- 仅依赖问题描述比结合模式解释更有效
日志代码在软件系统中用于记录关键事件,对调试和监控至关重要。但不安全的日志实践可能泄露敏感信息或引发日志注入攻击,威胁系统安全与隐私。现有研究多关注通用缺陷,缺乏对日志安全问题的系统分析,尤其未充分探索大语言模型(LLMs)在检测与修复中的应用。本文提出涵盖四类常见问题、十种具体模式的完整分类体系,并构建包含101个经人工审核标注的真实日志安全问题报告的基准数据集。进一步设计自动化框架,融合多种上下文知识评估LLMs在检测与修复日志安全问题上的能力。实验表明:尽管LLMs在检测上表现中等(平均准确率12.9%~52.5%),但在生成正确修复代码方面仍面临显著挑战。此外,仅使用问题描述即可提升检测准确率,优于仅提供安全模式解释或二者结合。整体结果为从业者提供可操作洞见,揭示当前大模型在安全日志领域的潜力与局限。
原文摘要 · Abstract (English)
Logging code plays an important role in software systems by recording key events and behaviors, which are essential for debugging and monitoring. However, insecure logging practices can inadvertently expose sensitive information or enable attacks such as log injection, posing serious threats to system security and privacy. Prior research has examined general defects in logging code, but systematic analysis of logging code security issues remains limited, particularly in leveraging LLMs for detection and repair. In this paper, we derive a comprehensive taxonomy of logging code security issues, encompassing four common issue categories and 10 corresponding patterns. We further construct a benchmark dataset with 101 real-world logging security issue reports that have been manually reviewed and annotated. We then propose an automated framework that incorporates various contextual knowledge to evaluate LLMs' capabilities in detecting and repairing logging security issues. Our experimental results reveal a notable disparity in performance: while LLMs are moderately effective at detecting security issues (e.g., the accuracy ranges from 12.9% to 52.5% on average), they face noticeable challenges in reliably generating correct code repairs. We also find that the issue description alone improves the LLMs' detection accuracy more than the security pattern explanation or a combination of both. Overall, our findings provide actionable insights for practitioners and highlight the potential and limitations of current LLMs for secure logging.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。