arXiv:2510.19670cs.CL2025-10

让边缘设备高效处理多模态数据,兼顾隐私与低延迟。

CoSense-LLM: Semantics at the Edge with Cost- and Uncertainty-Aware Cloud-Edge Cooperation

  • 用轻量编码将传感器数据转为可验证语义码,本地处理为主。
  • 边端响应延迟低于1秒(p95),带宽消耗降低,隐私数据不外传。
  • 适合对隐私、实时性要求高的智能家居、医疗等场景。

我们提出 CoSense-LLM,一种以边缘为核心的框架,将连续的多模态传感器流(如 Wi-Fi CSI、IMU、音频、RFID 及轻量视觉)转化为紧凑且可验证的语义标记,并在显式延迟、能耗、带宽和隐私约束下与大语言模型协同工作。系统包含四部分:(i) SenseFusion,一个轻量级编码器,对齐传感器嵌入并压缩为短离散代码序列;(ii) Edge-RAG,本地混合检索层,将生成结果锚定于特定场所策略与笔记;(iii) PromptRouter,基于成本与不确定性的决策策略,选择仅边端生成、边端加检索或紧凑云端升级;(iv) Secure Execution,可审计的数据最小化路径,确保原始波形不离开设备。系统支持分页/流式 KV 缓存、FlashAttention 风格内核、推测解码及量化 LoRA 适配器,并支持非独立同分布漂移下的本地个性化与联邦更新。在家庭、办公及诊所部署中,CoSense-LLM 实现了有依据的解释,满足严格服务等级目标:边端主导路径下端到端延迟低于 1 秒(p95),通过优先本地检索减少跨层级令牌与带宽开销,同时通过仅传输离散代码与脱敏元数据保障隐私。消融实验表明,Edge-RAG 提升事实一致性并减少矛盾,校准不确定性实现选择性回避与可控升级,KV 加速器降低每决策能耗。结果支持以边缘为先的设计,将语义理解、隐私保护与可预测延迟视为同等重要的目标,适用于干扰密集环境中的大模型部署。

原文摘要 · Abstract (English)

We present CoSense-LLM, an edge-first framework that turns continuous multimodal sensor streams (for example Wi-Fi CSI, IMU, audio, RFID, and lightweight vision) into compact, verifiable semantic tokens and coordinates with large language models under explicit latency, energy, bandwidth, and privacy constraints. CoSense-LLM has four parts: (i) SenseFusion, a lightweight encoder that aligns sensor embeddings with language and compresses them into short discrete code sequences; (ii) Edge-RAG, a local hybrid retrieval layer that grounds generation in site specific policies and notes; (iii) PromptRouter, a cost and uncertainty aware policy that selects edge only generation, edge plus retrieval, or compact cloud escalation; and (iv) Secure Execution, an auditable redaction path that enforces data minimization so raw waveforms never leave the device. The system works with modern serving optimizations, including paged or streaming KV caches, FlashAttention style kernels, speculative decoding, and quantized LoRA adapters, and supports on device personalization and federated updates under non IID drift. Across home, office, and clinic deployments, CoSense-LLM delivers grounded explanations while meeting tight service level objectives: it sustains sub second (p95) end to end latency on edge dominant paths, reduces inter tier token and bandwidth costs by preferring local retrieval grounded responses, and preserves privacy by transmitting only discrete codes and redacted metadata. Ablations show that Edge-RAG improves factual consistency and reduces contradictions, calibrated uncertainty enables selective abstention and controlled escalations, and KV plus decoding accelerators lower energy per decision. The results support an edge first design that treats semantics, privacy, and predictable latency as co equal goals for large model deployments in interference prone environments.

边缘计算多模态融合隐私保护大模型部署

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。