对比结构化与非结构化数据分类，发现复杂模型未必更优。
Challenges of Heterogeneity in Big Data: A Comparative Study of Classification in Large-Scale Structured and Unstructured Domains
- 用遗传算法和贝叶斯优化调参，结合Spark处理大规模文本
- 高维空间中线性模型优于深度网络；文本任务中简单模型更稳定
- 适合关注数据特性与算力限制的工程师和研究者
本研究通过对比结构化数据（Epsilon）与非结构化数据（Rest-Mex、IMDB）的分类策略，分析大数据中异构性（Variety）的影响。采用双方法：对数值数据使用Python中的遗传算法与Optuna进行进化与贝叶斯超参数优化；对大规模文本语料采用Apache Spark分布式处理。结果揭示“复杂性悖论”：在高维空间中，经优化的线性模型（如SVM、逻辑回归）表现优于深度架构与梯度提升模型；而在文本领域，分布式微调导致复杂模型过拟合，而基于Transformer的嵌入（ROBERTa）与贝叶斯目标编码等特征工程使简单模型实现良好泛化。本文提出一种基于数据特性与基础设施约束的统一算法选择框架。
原文摘要 · Abstract (English)
This study analyzes the impact of heterogeneity ("Variety") in Big Data by comparing classification strategies across structured (Epsilon) and unstructured (Rest-Mex, IMDB) domains. A dual methodology was implemented: evolutionary and Bayesian hyperparameter optimization (Genetic Algorithms, Optuna) in Python for numerical data, and distributed processing in Apache Spark for massive textual corpora. The results reveal a "complexity paradox": in high-dimensional spaces, optimized linear models (SVM, Logistic Regression) outperformed deep architectures and Gradient Boosting. Conversely, in text-based domains, the constraints of distributed fine-tuning led to overfitting in complex models, whereas robust feature engineering -- specifically Transformer-based embeddings (ROBERTa) and Bayesian Target Encoding -- enabled simpler models to generalize effectively. This work provides a unified framework for algorithm selection based on data nature and infrastructure constraints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。