构建电信标准表解读基准,揭示大模型在技术文档理解上的短板。
TeleTables: A Benchmark for Large Language Models in Telecom Table Interpretation
- 通过多阶段生成流程从3GPP标准中提取表格并构造问答对。
- 500个经人工验证的问答对显示小模型表理解能力差,大模型表现更优。
- 适合关注电信领域大模型应用与垂直领域微调的研究者使用。
大语言模型(LLMs)在电信工程任务、故障排查和复杂技术文档解析中日益受到关注,但现有研究显示其在3GPP标准等电信规范上表现不佳。我们指出,一个关键原因是这些标准密集包含表格以呈现核心信息,而现有模型对表格的隐式知识和显式解读能力尚未充分考察。为此,我们提出TeleTables,一个评估模型在技术规范中对表格的隐含知识与显式解读能力的基准。该基准通过新颖的多阶段数据生成流程构建,从3GPP标准中提取表格,并利用多模态与推理导向的LLMs生成并验证问题。最终公开的数据集包含500个经人工验证的问答对,每个对应多个格式的表格。评估表明,参数量低于100亿的小模型在回忆3GPP知识和解读表格方面均表现不佳,说明其预训练中缺乏对电信标准的覆盖以及对复杂技术材料的归纳偏差不足;而大模型展现出更强的表格推理能力。总体而言,TeleTables凸显了为可靠解读和推理电信标准而进行领域专用微调的必要性。
原文摘要 · Abstract (English)
Language Models (LLMs) are increasingly explored in the telecom industry to support engineering tasks, accelerate troubleshooting, and assist in interpreting complex technical documents. However, recent studies show that LLMs perform poorly on telecom standards, particularly 3GPP specifications. We argue that a key reason is that these standards densely include tables to present essential information, yet the LLM knowledge and interpretation ability of such tables remains largely unexamined. To address this gap, we introduce TeleTables, a benchmark designed to evaluate both the implicit knowledge LLMs have about tables in technical specifications and their explicit ability to interpret them. TeleTables is built through a novel multi-stage data generation pipeline that extracts tables from 3GPP standards and uses multimodal and reasoning-oriented LLMs to generate and validate questions. The resulting dataset, which is publicly available, comprises 500 human-verified question-answer pairs, each associated with the corresponding table in multiple formats. Our evaluation shows that, smaller models (under 10B parameters) struggle both to recall 3GPP knowledge and to interpret tables, indicating the limited exposure to telecom standards in their pretraining and the insufficient inductive biases for navigating complex technical material. Larger models, on the other hand, show stronger reasoning on table interpretation. Overall, TeleTables highlights the need for domain-specialized fine-tuning to reliably interpret and reason over telecom standards.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。