让大模型更懂表格结构，用低秩适配提升效率
TableLoRA: Low-rank Adaptation on Table Structure Understanding for Large Language Models
- 设计专用模块，用特殊标记和2D低秩矩阵编码表格位置信息
- 在4个数据集上优于普通LoRA和多种表格编码方法
- 适合需要高效处理表格的场景，尤其参数受限时
表格数据在众多领域至关重要，而大语言模型（LLMs）在高参数效率范式下理解表格数据仍具挑战，尤其体现在表格序列化与二维结构信息在单维序列中的表示。为此，我们提出TableLoRA，一种专用于提升LLM在参数高效微调（PEFT）中表格结构理解能力的模块。该模块采用特殊标记进行表格序列化，并通过2D LoRA编码单元格位置的低秩信息。在四个表格相关数据集上的实验表明，TableLoRA始终优于原始LoRA，并超越多种对比的表格编码方法。结果表明，TableLoRA作为面向表格的低秩适配器，显著增强了LLM在低参数设置下处理表格数据的能力，展现出在表格任务中的强大潜力。
原文摘要 · Abstract (English)
Tabular data are crucial in many fields and their understanding by large language models (LLMs) under high parameter efficiency paradigm is important. However, directly applying parameter-efficient fine-tuning (PEFT) techniques to tabular tasks presents significant challenges, particularly in terms of better table serialization and the representation of two-dimensional structured information within a one-dimensional sequence. To address this, we propose TableLoRA, a module designed to improve LLMs' understanding of table structure during PEFT. It incorporates special tokens for serializing tables with special token encoder and uses 2D LoRA to encode low-rank information on cell positions. Experiments on four tabular-related datasets demonstrate that TableLoRA consistently outperforms vanilla LoRA and surpasses various table encoding methods tested in control experiments. These findings reveal that TableLoRA, as a table-specific LoRA, enhances the ability of LLMs to process tabular data effectively, especially in low-parameter settings, demonstrating its potential as a robust solution for handling table-related tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。