arXiv:2411.15211cs.LGcs.AI2024-11被引 15

用轻量微调让大模型精准感知光照,三类任务表现远超现有方法。

LightLLM: A Versatile Large Language Model for Predictive Light Sensing

  • 融合传感器数据与环境提示,通过轻量组件微调冻结的LLM。
  • 光照定位精度提升4.4倍,室内太阳辐射估测提升3.4倍。
  • 适合需低资源适配的智能传感场景,如城市照明、光伏预测。

我们提出LightLLM,一种针对光传感任务微调预训练大语言模型(LLM)的模型。它集成传感器数据编码器提取关键特征、上下文提示提供环境信息,并通过融合层生成统一表征。该表征输入冻结的预训练LLM,仅通过添加轻量可训练组件实现微调,无需修改原始参数,从而在极低计算开销和重训成本下适应新任务。我们在三种光传感任务中实现该模型:基于光的定位、户外太阳能预测和室内太阳辐射估计。使用真实实验数据集验证,LightLLM显著优于当前最优方法,在未见环境中定位精度提升4.4倍,室内太阳辐射估测提升3.4倍。此外,其性能超越直接提示的ChatGPT-4,凸显了专用架构在传感器数据与文本提示融合上的优势。

原文摘要 · Abstract (English)

We propose LightLLM, a model that fine tunes pre-trained large language models (LLMs) for light-based sensing tasks. It integrates a sensor data encoder to extract key features, a contextual prompt to provide environmental information, and a fusion layer to combine these inputs into a unified representation. This combined input is then processed by the pre-trained LLM, which remains frozen while being fine-tuned through the addition of lightweight, trainable components, allowing the model to adapt to new tasks without altering its original parameters. This approach enables flexible adaptation of LLM to specialized light sensing tasks with minimal computational overhead and retraining effort. We have implemented LightLLM for three light sensing tasks: light-based localization, outdoor solar forecasting, and indoor solar estimation. Using real-world experimental datasets, we demonstrate that LightLLM significantly outperforms state-of-the-art methods, achieving 4.4x improvement in localization accuracy and 3.4x improvement in indoor solar estimation when tested in previously unseen environments. We further demonstrate that LightLLM outperforms ChatGPT-4 with direct prompting, highlighting the advantages of LightLLM's specialized architecture for sensor data fusion with textual prompts.

大模型光传感轻量微调多模态融合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。