评测LLM生成结构化输出的正确性与碳排放,发现紧凑格式更省电但易出错。
Are LLMs Ready for TOON? Benchmarking Structural Correctness-Sustainability Trade-offs in Novel Structured Output Formats
- 引入碳效率评估框架,综合考量输出格式的正确性与能耗。
- TOON格式输出更紧凑、碳排放更低,但结构正确性较差。
- 模型越大,格式间差距越小,适合重视低碳的部署场景。
大型语言模型(LLMs)在生成机器可读的结构化输出方面日益重要。现有基准主要关注输出的结构正确性,却忽视了不同输出格式的推理环境影响。本文提出一种兼顾可持续性的评估框架,测量标记使用量、生成时间和估算的碳排放。在此框架下,我们引入环境感知生成正确性得分(GCS_env),将结构正确性与碳效率统一衡量。系统性地将新型TOON格式与传统格式(JSON、XML、YAML)在多种架构和参数规模的LLMs上进行对比。结果显示存在显著权衡:无原生支持时,TOON输出更紧凑、碳排放更低,但结构正确性下降;模型容量越大,该差距越小。环境感知评分可因部署目标改变格式排名,强调需纳入可持续性考量,并为大规模碳意识部署中采用紧凑表示如TOON提供实证支持。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly required to generate structured, machine-readable outputs for downstream systems. While recent benchmarks have focused on evaluating the structural correctness of such outputs, the environmental impact of inference for different output formats has largely been overlooked. In this paper, we argue that structured output formats should be assessed not only in terms of correctness, but also with respect to their environmental efficiency. To this end, we introduce a sustainability-aware evaluation framework for structured generation that measures token usage, generation time, and estimated carbon emissions. Within this framework, we propose the Environment-Aware Generation Correctness Score (GCS_env), a unified metric that integrates structural correctness with carbon-aware efficiency. Using this framework, we systematically benchmark the novel TOON format against established representations (JSON, XML, YAML) across multiple LLMs spanning different architectures and parameter scales. Our results reveal a consistent trade-off: TOON yields markedly more compact outputs and lower emissions, but lower structural correctness when models lack native support. We show that increased model capacity reduces this gap and that environment-aware scoring can shift format rankings depending on deployment priorities. highlighting the need for sustainability-inclusive benchmarking and provides empirical evidence that compact representations such as TOON can offer practical advantages in large-scale, carbon-conscious LLM deployments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。