用大模型检测物联网规则冲突,发现其结构推理能力不足。
Cracking IoT Security: Can LLMs Outsmart Static Analysis Tools?
- 用大模型分析物联网规则间的语义与结构依赖关系
- 模型在跨规则结构推理上准确率显著下降,尤其面对规则变形时
- 适合研究人机协同安全检测工具的开发者参考
智能家居物联网平台如openHAB依赖触发-动作-条件(TAC)规则自动化设备行为,但规则间的交互可能引发隐性依赖、冲突触发或重叠条件等交互威胁。传统上依赖符号化、约束驱动的静态分析进行威胁识别。本文首次对大语言模型(LLMs)在多类别交互威胁分类体系下的表现进行全面评估,测试对象包括原始openHAB(oHC/IoTB)数据集和一个针对规则变换设计的结构挑战型突变数据集(Mutation dataset)。我们对比了Llama 3.1 8B、Llama 70B、GPT-4o、Gemini-2.5-Pro和DeepSeek-R1在零样本、单样本及双样本设置下的表现,并与oHIT的手动验证真值进行比较。结果显示,尽管大模型在动作和条件相关威胁上表现出良好的语义理解能力,但在需要跨规则结构推理的任务中,其准确率显著下降,尤其在规则被改写后。不同模型在各类威胁和提示设置下表现差异大,无一模型具备一致性可靠性。相比之下,符号推理基线在两个数据集上均保持稳定检测能力,不受规则重写或结构扰动影响。这表明当前大模型尚不足以独立承担物联网环境中关键安全威胁的检测任务。本文讨论了对工具设计的启示,并提出将符号分析与大模型语义理解结合的混合架构,有望降低误报率同时保持结构严谨性。
原文摘要 · Abstract (English)
Smart home IoT platforms such as openHAB rely on Trigger Action Condition (TAC) rules to automate device behavior, but the interplay among these rules can give rise to interaction threats, unintended or unsafe behaviors emerging from implicit dependencies, conflicting triggers, or overlapping conditions. Identifying these threats requires semantic understanding and structural reasoning that traditionally depend on symbolic, constraint-driven static analysis. This work presents the first comprehensive evaluation of Large Language Models (LLMs) across a multi-category interaction threat taxonomy, assessing their performance on both the original openHAB (oHC/IoTB) dataset and a structurally challenging Mutation dataset designed to test robustness under rule transformations. We benchmark Llama 3.1 8B, Llama 70B, GPT-4o, Gemini-2.5-Pro, and DeepSeek-R1 across zero-, one-, and two-shot settings, comparing their results against oHIT's manually validated ground truth. Our findings show that while LLMs exhibit promising semantic understanding, particularly on action- and condition-related threats, their accuracy degrades significantly for threats requiring cross-rule structural reasoning, especially under mutated rule forms. Model performance varies widely across threat categories and prompt settings, with no model providing consistent reliability. In contrast, the symbolic reasoning baseline maintains stable detection across both datasets, unaffected by rule rewrites or structural perturbations. These results underscore that LLMs alone are not yet dependable for safety critical interaction-threat detection in IoT environments. We discuss the implications for tool design and highlight the potential of hybrid architectures that combine symbolic analysis with LLM-based semantic interpretation to reduce false positives while maintaining structural rigor.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。