提出6维数据特征框架,系统分析传感器大数据难题。
A systematic data characteristic understanding framework towards physical-sensor big data challenges
- 基于6Vs模型构建数据驱动的特征分析框架
- 量化识别体积、速度、真实性等维度挑战
- 适用于物联网传感器数据预处理优化
大数据为现代社会带来新机遇,也给数据科学家带来挑战。传感器网络与物联网的普及导致物理传感器数据规模空前。然而,高质量大数据分析面临诸多困难。为揭示数据挑战并提升数据质量,必须定量剖析数据特征。现有研究缺乏对时间相关特征的分析。通过全生命周期的大数据分析效率与精度提升,需全面理解数据特征以应对隐藏挑战。为此,本文提出基于6Vs模型的系统性数据特征理解框架,从数据体量、多样性、速度、真实性、价值和可变性六个维度,通过一组统计指标揭示数据特征。该模型仅依赖数据驱动指标,提高分析客观性,并包含物理传感器数据的时间特征指标。同时将大数据挑战与6Vs各维度关联,实现定量理解。最后构建实施流程,开展两个案例研究,展示物理传感器数据特征理解过程及数据预处理建议。该框架可分析所有物理传感器数据,识别后续分析中的潜在挑战,并提供预处理推荐。
原文摘要 · Abstract (English)
Big data present new opportunities for modern society while posing challenges for data scientists. Recent advancements in sensor networks and the widespread adoption of IoT have led to the collection of physical-sensor data on an enormous scale. However, significant challenges arise in high-quality big data analytics. To uncover big data challenges and enhance data quality, it is essential to quantitatively unveil data characteristics. Furthermore, the existing studies lack analysis of the specific time-related characteristics. Enhancing the efficiency and precision of data analytics through the big data lifecycle requires a comprehensive understanding of data characteristics to address the hidden big data challenges. To fill in the research gap, this paper proposes a systematic data characteristic framework based on a 6Vs model. The framework aims to unveil the data characteristics in terms of data volume, variety, velocity, veracity, value, and variability through a set of statistical indicators. This model improves the objectivity of data characteristic understanding by relying solely on data-driven indicators. The indicators related to time-related characteristics in physical-sensor data are also included. Furthermore, the big data challenges are linked to each dimension of the 6Vs model to gain a quantitative understanding of the data challenges. Finally, a pipeline is developed to implement the proposed framework, and two case studies are conducted to illustrate the process of understanding the physical-sensor data characteristics and making recommendations for data preprocessing to address the big data challenges. The proposed framework is able to analyze the characteristics of all physical-sensor data, therefore, identifying potential challenges in subsequent analytics, and providing recommendations for data preprocessing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。