用显式指令提升网络安全实体识别准确率,仅需1%标注数据即可媲美全量微调。
From Retrieval to Reasoning: A Framework for Cyber Threat Intelligence NER with Explicit and Adaptive Instructions
- 将威胁情报概念转为分层指令:策略、技巧、流程
- 自适应优化标注规范,仅用1%数据就达到全量微调效果
- 在5个基准上超越检索方法,复杂任务性能提升超10%
网络安全情报(CTI)自动化依赖命名实体识别(NER)从非结构化文本中提取关键信息。当前主流方法依赖大语言模型(LLM)的上下文学习(ICL),但本研究发现其成功主要源于检索样本中实体类型偶然重叠,而非语义相似性,暴露了隐式归纳的不可靠性。为此,我们提出TTPrompt框架,将战术、技术与规程(TTPs)转化为显式指令层级:战术对应任务定义,技巧对应引导策略,规程对应标注指南。针对静态指南适应性差的问题,引入反馈驱动的指令精炼(FIR),使LLM能基于少量标注数据自我修正指南,适配不同标注风格。在五个CTI NER基准上的实验表明,TTPrompt持续优于检索基线。例如,在LADDER上微调后达到71.96%的微观F1,接近全量微调基线;在复杂数据集CTINexus上,宏观F1超过全量微调的ACLM模型10.91个百分点。
原文摘要 · Abstract (English)
The automation of Cyber Threat Intelligence (CTI) relies heavily on Named Entity Recognition (NER) to extract critical entities from unstructured text. Currently, Large Language Models (LLMs) primarily address this task through retrieval-based In-Context Learning (ICL). This paper analyzes this mainstream paradigm, revealing a fundamental flaw: its success stems not from global semantic similarity but largely from the incidental overlap of entity types within retrieved examples. This exposes the limitations of relying on unreliable implicit induction. To address this, we propose TTPrompt, a framework shifting from implicit induction to explicit instruction. TTPrompt maps the core concepts of CTI's Tactics, Techniques, and Procedures (TTPs) into an instruction hierarchy: formulating task definitions as Tactics, guiding strategies as Techniques, and annotation guidelines as Procedures. Furthermore, to handle the adaptability challenge of static guidelines, we introduce Feedback-driven Instruction Refinement (FIR). FIR enables LLMs to self-refine guidelines by learning from errors on minimal labeled data, adapting to distinct annotation dialects. Experiments on five CTI NER benchmarks demonstrate that TTPrompt consistently surpasses retrieval-based baselines. Notably, with refinement on just 1% of training data, it rivals models fine-tuned on the full dataset. For instance, on LADDER, its Micro F1 of 71.96% approaches the fine-tuned baseline, and on the complex CTINexus, its Macro F1 exceeds the fine-tuned ACLM model by 10.91%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。