JTON通过分列编码减少结构化数据的冗余，提升大模型处理效率。
JTON: A Token-Efficient JSON Superset with Zen Grid Tabular Encoding for Large Language Models
- 将表格列头统一编码，用分号分隔值，减少重复键名
- 跨7个领域平均节省28.5%令牌，最多降60%
- 兼容JSON类型系统，适合需高效处理表格的LLM应用
当大语言模型处理结构化数据时，序列化格式直接影响成本与上下文利用率。标准JSON在表格数组中重复键名，开销随行数线性增长。本文提出JTON（JSON表对象表示法），其核心思想是Zen Grid：将列头合并为单行，用分号编码数值，保留JSON类型系统的同时消除冗余。在7个真实领域中，Zen Grid相较JSON紧凑版平均减少28.5%令牌（裸字符串场景达32%）；10个LLM的理解测试显示，相比JSON净增0.3个百分点准确率：4个模型提升，3个持平，3个小幅下降。12个LLM的生成测试在少样本与零样本设置下均实现100%语法正确性。基于Rust/PyO3的参考实现支持SIMD加速解析，速度达Python json模块的1.4倍。代码、683向量测试集及所有实验数据均已公开。
原文摘要 · Abstract (English)
When LLMs process structured data, the serialization format directly affects cost and context utilization. Standard JSON wastes tokens repeating key names in every row of a tabular array--overhead that scales linearly with row count. This paper presents JTON (JSON Tabular Object Notation), a strict JSON superset whose main idea, Zen Grid, factors column headers into a single row and encodes values with semicolons, preserving JSON's type system while cutting redundancy. Across seven real-world domains, Zen Grid reduces token counts by 15-60% versus JSON compact (28.5% average; 32% with bare_strings). Comprehension tests on 10 LLMs show a net +0.3 pp accuracy gain over JSON: four models improve, three hold steady, and three dip slightly. Generation tests on 12 LLMs yield 100% syntactic validity in both few-shot and zero-shot settings. A Rust/PyO3 reference implementation adds SIMD-accelerated parsing at 1.4x the speed of Python's json module. Code, a 683-vector test suite, and all experimental data are publicly available.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。