用物联网数据提升大模型对现实世界的推理能力
IoT-LLM: a framework for enhancing Large Language Model reasoning from real-world sensor data
- 将传感器数据预处理后注入大模型，增强其感知能力
- 在真实任务上使模型推理准确率提升49.4%
- 适合需要物理世界理解的智能系统开发者
大语言模型在文本任务中表现优异，但在物理世界推理任务中常显不足。受人类认知启发——感知是推理的基础，本文探索通过物联网（IoT）数据与相关知识增强大模型的感知能力。系统研究了大模型处理物联网传感任务的能力，并提出统一框架IoT-LLM以提升该能力。该框架包含三个步骤：将物联网数据预处理为适配格式、通过面向物联网的检索增强生成扩展模型知识、利用思维链提示激活模型常识知识。设计了一个包含五个真实世界任务的基准，涵盖不同数据类型与推理复杂度。实验结果表明，IoT-LLM显著提升大模型在物联网传感任务上的推理性能，如GPT-4o-mini在多项任务上平均提升49.4%。
原文摘要 · Abstract (English)
Large Language Models excel in textual tasks but often struggle with physical-world reasoning tasks. Inspired by human cognition, where perception is fundamental to reasoning, we explore augmenting LLMs with enhanced perception abilities using Internet of Things (IoT) data and pertinent knowledge. In this work, we systematically study LLMs' capability to address IoT-sensory tasks by augmenting their perception and knowledge base, and then propose a unified framework, IoT-LLM, to enhance such capability. In IoT-LLM, we customize three steps: preprocessing IoT data into suitable formats, expanding LLMs knowledge via IoT-oriented retrieval-augmented generation and activating LLMs commonsense knowledge through chain-of-thought prompting. We design a benchmark comprising five real-world tasks with varying data types and reasoning complexities to evaluate the performance of IoT-LLM. Experimental results reveal that IoT-LLM significantly improves the performance of IoT-sensory task reasoning of LLMs, with models like GPT-4o-mini showing a 49.4% average improvement over previous methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。