构建开源网络安全问答数据集，提升安全分析效率。
AttackQA: Development and Adoption of a Dataset for Assisting Cybersecurity Operations using Fine-tuned and Open-Source LLMs
- 用轻量级开源模型生成2.5万条安全问答对，配合理由增强可解释性。
- 微调开源模型在问答准确率上超越GPT-4o，实现更高性价比。
- 全流程开源，支持高速安全分析，适合安全运维人员使用。
针对网络安全运营中心（SOC）中分析师的响应需求，本文构建了一个名为AttackQA的专用问答数据集，包含25,335组问题与答案对，并附有推理过程以支持模型微调与评估。其中80%的数据由轻量级开源模型LLama 3 8B生成，在SambaNova SN40L硬件上每秒生成超过1100个令牌，保持16位精度。为保障质量，采用微调后的LLama 3 70B模型识别并剔除低质量样本。在构建基于RAG的问答系统时，实验表明微调开源嵌入与语言模型的性能优于OpenAI的GPT-4o。此外，使用Llama 3.1 405B作为评判器评估答案正确性，实现了全开源、高速的RAG与评估流水线，并提供模型准确性基准。
原文摘要 · Abstract (English)
Retrieval-augmented generation (RAG) on specialized domain datasets has shown improved performance when large language models (LLMs) are fine-tuned for generating responses to user queries. In this study, we develop a cybersecurity question-answering (Q\&A) dataset, called AttackQA, and employ it to build a RAG-based Q\&A system designed for analysts in security operations centers. The dataset comprises 25,335 Q\&A pairs, accompanied by rationales to facilitate fine-tuning and evaluation. 80\% of the dataset was generated with help of a lightweight open-source LLM (LLama 3 8B), which produced over 1100 tokens per second with full 16-bit precision on SambaNova System's SN40L specialized hardware. To ensure dataset quality, we fine-tuned LLama 3 70B to detect and reject low-quality Q\&A pairs. In using the dataset for RAG, we demonstrate that fine-tuning open-source embeddings and LLMs can yield superior accuracy compared to OpenAI's state-of-the-art proprietary embedding and LLM (GPT-4o). Furthermore, we use Llama 3.1 405B as a judge to evaluate answer correctness, enabling the creation of a fully open-source, high-speed RAG and evaluation pipeline with a benchmark for model accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。