将表格数据转为适合大模型训练的统一序列格式
Innovative tokenisation of structured data for LLM training
- 用固定标记+可学习子词编码混合表示结构化数据
- 处理3100万条网络流，压缩比达6.18:1
- 适合想训练结构数据大模型的研究者
数据表示仍是机器学习中的核心挑战，尤其在将基于序列的Transformer和大语言模型（LLM）应用于结构化表格数据时。现有方法难以有效编码数值与类别特征的混合，也难以保留表格的固有结构。本文提出一种新型混合分词方法，将表格数据转换为统一的序列格式，适用于LLM训练。该方法结合预定义的固定标记（用于结构元素和低基数类别特征）与基于字节对编码（BPE）的可学习子词词汇表（用于高基数和连续值）。我们通过在大规模NetFlow数据集CIDDS-001上的应用，构建了一个用于网络入侵检测系统（NIDS）基础模型的语料库。评估表明，该方法高效可靠：在不到五小时内处理超过3100万条网络流，实现6.18:1的数据压缩比，生成一个超十亿令牌的计算可行语料库，为结构化数据训练基础模型提供了一条通用且可行的路径。
原文摘要 · Abstract (English)
Data representation remains a fundamental challenge in machine learning, particularly when adapting sequence-based architectures like Transformers and Large Language Models (LLMs) for structured tabular data. Existing methods often fail to cohesively encode the mix of numerical and categorical features or preserve the inherent structure of tables. This paper introduces a novel, hybrid tokenisation methodology designed to convert tabular data into a unified, sequential format suitable for LLM training. Our approach combines predefined fixed tokens to represent structural elements and low-cardinality categorical features, with a learned subword vocabulary using Byte-Pair Encoding (BPE) for high-cardinality and continuous values. We demonstrate the efficacy of this technique by applying it to a large-scale NetFlow dataset (CIDDS-001), preparing a corpus for a Network Intrusion Detection System (NIDS) foundation model. The evaluation shows that our method is highly efficient, processing over 31 million network flows in under five hours and achieving a significant data compression ratio of 6.18:1. This process resulted in a computationally manageable corpus of over one billion tokens, establishing a viable and generalisable pathway for training foundation models on structured data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。