评测大模型处理深层嵌套JSON数据的能力,填补真实网络信息提取的评估空白。
DeepJSONEval: Benchmarking Complex Nested JSON Data Mining for Large Language Models
- 构建2100个跨领域复杂嵌套JSON样本,按难度分级
- 大模型在深层结构解析上表现差异显著,最高错误率超60%
- 适合研究结构化生成、信息抽取与大模型评估的开发者
互联网充斥着低密度、高冗余的信息,如社交媒体评论、重复新闻和长篇讨论,难以高效提取价值洞察。多层嵌套JSON结构通过将信息压缩为语义丰富的层次化表示,有效解决此问题,其以键值对、数组和嵌套对象组织数据,保留上下文关系,支持高效存储、检索与语义查询。例如,在新闻聚合中,可将文章元数据(标题、作者、日期)、内容(文本、多媒体)及多媒体信息(类型、说明)分层嵌套。大语言模型(LLMs)在网页数据挖掘中发挥变革性作用,能直接将非结构化文本解析为复杂JSON模式的结构化输出。然而,现有评估基准过度强调纯JSON生成能力,忽视了数据理解与提取能力的评估,与实际应用脱节。为此,我们提出DeepJSONEval,一个包含2100个跨领域实例、具有深度嵌套结构并按难度分类的新基准。实验表明,不同LLMs在处理此类复杂结构时存在显著性能差距。该基准与数据集已开源,以推动结构化JSON生成研究。(https://github.com/GTS-AI-Infra-Lab-SotaS/DeepJSONEval)
原文摘要 · Abstract (English)
The internet is saturated with low-density, high-redundancy information, such as social media comments, repetitive news, and lengthy discussions, making it difficult to extract valuable insights efficiently. Multi-layer nested JSON structures provide an effective solution by compressing such information into semantically rich, hierarchical representations, which organize data into key-value pairs, arrays, and nested objects, preserving contextual relationships and enabling efficient storage, retrieval, and semantic querying. For instance, in news aggregation, a JSON object can nest an article's metadata (title, author, date), content (text, multimedia), and multimedia information (multimedia type, caption) hierarchically. Large Language Models (LLMs) play a transformative role in web data mining by parsing unstructured text and outputting structured results directly into complex JSON schemas. However, current benchmarks for evaluating LLMs' JSON output capabilities overemphasize pure JSON generation rather than assessing data comprehension and extraction abilities, a limitation that lacks relevance to practical web data mining tasks. To address this, we introduce DeepJSONEval, a novel benchmark featuring 2100 multi-domain instances with deep nested structures, categorized by difficulty. Experiments show significant performance gaps among LLMs in handling such complexity. Our benchmark and datasets are open-sourced to advance research in structured JSON generation.(https://github.com/GTS-AI-Infra-Lab-SotaS/DeepJSONEval).
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。