arXiv:2607.18725cs.CLcs.AI2026-07

先诊断再微调,避免小模型在网络安全问答中知识丢失。

Find Before You Fine-Tune: A Diagnostic Study of Small LLMs for Cybersecurity QA

论文配图:Find Before You Fine-Tune: A Diagnostic Study of Small LLMs for Cybersecurity QA
图 1 · 摘自论文原文
  • 提出FiT框架,评估小模型在词汇、参数知识和上下文理解三方面能力。
  • 微调后模型知识普遍下降,指令导向微调导致知识崩溃且排名反转。
  • 诊断得分可预判微调效果,帮助筛选适合的模型,节省成本。

大型语言模型(LLMs)正被越来越多地用于关键领域问答任务,但选择适配的小模型前仍面临挑战。微调虽能提升领域对齐,却可能损害已有知识、削弱指令遵循能力或加剧幻觉,尤其在标签数据稀缺或快速演变的网络安全场景下。本文提出FiT(Find before Fine-Tune)——一种面向任务的诊断框架,从词汇识别、参数化知识和检索信息上下文化三个维度评估70亿参数级开源小模型。在两种微调策略下对五款模型进行实证研究发现:微调并非普遍有益,反而持续损害词汇与参数知识;以知识为导向的微调仅造成适度降级且排名保持稳定,而以指令为导向的微调则引发知识崩溃并导致知识排名反转,同时保留检索驱动的上下文理解能力。通过秩相关分析量化这些模式,证实预微调的FiT得分可预测微调后的变化方向。结果表明,任务导向诊断有助于淘汰不适用模型,避免无效微调,支持更安全的小模型部署于网络安全问答流程。

原文摘要 · Abstract (English)

Large Language Models (LLMs) are increasingly fine-tuned for critical-domain Question-Answering (QA), yet choosing which small model to adapt, before paying the cost of adaptation, remains difficult. Fine-tuning can improve domain alignment, but it may also erode prior knowledge, weaken instruction-following, or increase hallucination, especially when labeled data are scarce or rapidly evolving as in cybersecurity. We present FiT (Find before Fine-Tune), a task-oriented diagnostic framework that characterizes small LLMs along three capabilities required for cybersecurity QA: vocabulary recognition, parametric knowledge, and contextualization of retrieved information. Using FiT, we conduct an empirical study of five open-weight 7-billion-parameter models under two fine-tuning regimes. We find that fine-tuning does not uniformly help: it consistently degrades vocabulary and parametric knowledge in small models, and the two regimes trade off differently. Knowledge-focused tuning causes moderate, rank-preserving degradation, whereas instruction-focused tuning collapses measured knowledge through induced abstention, inverting the knowledge ranking while leaving retrieval-grounded contextualization essentially intact. We quantify these regime-specific patterns with rank-correlation analysis and show that pre-fine-tuning FiT scores anticipate the direction of post-tuning change. Our results suggest that task-oriented diagnosis can screen out unsuitable models, avoid unnecessary fine-tuning, and support safer deployment of small LLMs in cybersecurity QA pipelines.

小模型网络安全微调诊断知识保持

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。