让大模型原生生成符合复杂结构的自洽对象，无需提示工程。
Structured Object Language Modeling (SoLM): Native Structured Objects Generation Conforming to Complex Schemas with Self-Supervised Denoising
- 将结构化对象生成视为语言建模任务，用自监督去噪训练模型。
- 在真实数据集上训练后，性能超越需提示工程的顶尖大模型。
- 适合需要高效、自动结构化数据生成的场景，如知识库构建。
本文研究如何生成符合复杂模式的结构化对象，其不同组件（属性、字段、列、特性）间存在复杂依赖关系。对象成分可为短结构化事实或长自然语言描述，需在冗余信息中保持相对一致，并与世界知识保持绝对一致。我们将其建模为结构化对象语言建模（SoLM）问题，训练大模型原生完成该任务，无需提示工程。提出自监督去噪方法，从已有结构化对象数据集中学习。输入可为原始对象（用于修复、补全、归一化），也可为非结构化文本（用于结构化转换）。实验表明，该方法提供强基线，结合少量人工示范的监督微调可进一步提升性能。结果表明，其效果匹配或优于需提示工程的前沿大模型（Claude 3、Mixtral-8x7B），且成本低一个数量级。
原文摘要 · Abstract (English)
In this paper, we study the problem of generating structured objects that conform to a complex schema, with intricate dependencies between the different components (facets) of the object. The facets of the object (attributes, fields, columns, properties) can be a mix of short, structured, type-constrained facts, or long natural-language descriptions. The object has to be self-consistent between the different facets in the redundant information it carries (relative consistency), while being grounded with respect to world knowledge (absolute consistency). We frame the problem as a Language Modeling problem (Structured Object Language Modeling) and train an LLM to perform the task natively, without requiring instructions or prompt-engineering. We propose a self-supervised denoising method to train the model from an existing dataset of such objects. The input query can be the existing object itself, in which case the model acts as a regenerator, completing, correcting, normalizing the input, or any unstructured blurb to be structured. We show that the self-supervised denoising training provides a strong baseline, and that additional supervised fine-tuning with small amount of human demonstrations leads to further improvement. Experimental results show that the proposed method matches or outperforms prompt-engineered general-purpose state-of-the-art LLMs (Claude 3, Mixtral-8x7B), while being order-of-magnitude more cost-efficient.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。