arXiv:2412.17348cs.LG2024-12被引 2

用生成式Transformer直接处理嵌套键值数据,保持层级语义。

ORIGAMI: A generative transformer architecture for predictions from semi-structured data

  • 通过保留结构的分词器和键值位置编码,直接解析JSON等半结构化数据。
  • 在多个真实数据集上优于传统模型,在代码分类任务中超越卷积与图神经网络。
  • 无需修改架构即可处理单标签和多标签任务,适合需要端到端建模的场景。

尽管JSON等半结构化数据格式广泛使用,但将其直接用于端到端监督学习的研究仍较匮乏。本文提出ORIGAMI(Object RepresentatIon via Generative Autoregressive ModellIng),一种基于Transformer的架构,可直接处理嵌套的键/值对,并保留其层级语义。关键技术包括:(1) 结构保持的分词器,(2) 创新的键/值位置编码方案,(3) 语法约束的训练与推理框架,确保输出合法性并加速收敛。该设计实现高效的半结构化数据端到端建模。通过将分类重构为下一个词预测任务,ORIGAMI无需修改架构即可支持单标签与多标签任务。在多种领域评估中表现优异:在转换为JSON的标准表格基准上,性能与经典及前沿方法相当;在原生JSON数据集上,多标签分类任务超越基线,在代码分类任务中优于卷积神经网络与图神经网络等专用模型。大量消融实验验证了各组件的有效性,确立了ORIGAMI作为半结构化数据端到端学习的稳健框架。

原文摘要 · Abstract (English)

Despite the popularity and widespread use of semi-structured data formats such as JSON, end-to-end supervised learning applied directly to such data remains underexplored. We present ORIGAMI (Object RepresentatIon via Generative Autoregressive ModellIng), a transformer-based architecture that directly processes nested key/value pairs while preserving their hierarchical semantics. Our key technical contributions include: (1) a structure-preserving tokenizer, (2) a novel key/value position encoding scheme, and (3) a grammar-constrained training and inference framework that ensures valid outputs and accelerates training convergence. These enhancements enable efficient end-to-end modeling of semi-structured data. By reformulating classification as next-token prediction, ORIGAMI naturally handles both single-label and multi-label tasks without architectural modifications. Empirical evaluation across diverse domains demonstrates ORIGAMI's effectiveness: On standard tabular benchmarks converted to JSON, ORIGAMI remains competitive with classical and state-of-the-art approaches. On native JSON datasets, we outperform baselines on multi-label classification and specialized models such as convolutional and graph neural networks on a code classification task. Through extensive ablation studies, we validate the impact of each architectural component and establish ORIGAMI as a robust framework for end-to-end learning on semi-structured data.

生成模型半结构化数据Transformer

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。