让大模型只关注重要特征关系，提升表格生成质量
Not All Features Deserve Attention: Graph-Guided Dependency Learning for Tabular Data Generation with Language Models
- 用外部依赖图指导注意力，筛选关键特征交互
- 在复杂数据集上生成质量提升12%，合成数据表现优异
- 轻量改造不侵入原模型，适合结构敏感的表格生成
大型语言模型（LLMs）在建模文本化的特征-值对方面展现出强大潜力，但表格数据普遍存在稀疏的特征级依赖关系，许多特征交互在结构上并不重要。而LLM的自注意力机制会无差别地分配关注，导致关键关系被稀释，尤其在依赖复杂或语义模糊的特征时问题更明显。为此，我们提出GraDe（Graph-Guided Dependency Learning），通过将外部提取的功能依赖图引入LLM的注意力机制，显式地强化关键特征交互，抑制无关关系。GraDe采用轻量级动态图学习模块，在多种真实世界数据集上的实验表明，其在复杂数据集上相较现有方法最高提升12%；在合成数据质量上也达到先进水平。该方法侵入性极小，为结构感知的表格数据生成提供了实用解决方案。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have shown strong potential for tabular data generation by modeling textualized feature-value pairs. However, tabular data inherently exhibits sparse feature-level dependencies, where many feature interactions are structurally insignificant. This creates a fundamental mismatch as LLMs' self-attention mechanism inevitably distributes focus across all pairs, diluting attention on critical relationships, particularly in datasets with complex dependencies or semantically ambiguous features. To address this limitation, we propose GraDe (Graph-Guided Dependency Learning), a novel method that explicitly integrates sparse dependency graphs into LLMs' attention mechanism. GraDe employs a lightweight dynamic graph learning module guided by externally extracted functional dependencies, prioritizing key feature interactions while suppressing irrelevant ones. Our experiments across diverse real-world datasets demonstrate that GraDe outperforms existing LLM-based approaches by up to 12% on complex datasets while achieving competitive results with state-of-the-art approaches in synthetic data quality. Our method is minimally intrusive yet effective, offering a practical solution for structure-aware tabular data modeling with LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。