为文本简化错误分析提供新分类体系和标注数据集
Resource for Error Analysis in Text Simplification: New Taxonomy and Test Collection
- 提出信息扭曲导向的错误分类体系
- 构建科学文本自动简化后的人工标注数据集
- 支持错误检测与分类模型评估，助力提升简化质量
普通公众常接触复杂文本却缺乏时间或专业知识理解，导致误信息传播。自动文本简化（ATS）有助于提升信息可读性，但其评估方法未能跟上大语言模型（LLMs）的发展，现有评测指标与实际错误存在脱节。人工检查揭示了多种错误类型，凸显了建立更精细评估框架的迫切需求。本文通过引入一个用于检测和分类简化文本错误的测试集来填补这一空白。首先，提出一种以信息扭曲为核心的错误分类体系；其次，构建了一个由自动生成的科学文本组成的平行语料库，并经过人工标注，标签基于所提分类体系；最后，分析数据集质量并评估现有模型在该分类体系下的错误检测与分类性能。这些贡献为研究人员提供了更有效的工具，以改进ATS中的错误评估、开发更可靠的模型，从而提升自动简化文本的整体质量。
原文摘要 · Abstract (English)
The general public often encounters complex texts but does not have the time or expertise to fully understand them, leading to the spread of misinformation. Automatic Text Simplification (ATS) helps make information more accessible, but its evaluation methods have not kept up with advances in text generation, especially with Large Language Models (LLMs). In particular, recent studies have shown that current ATS metrics do not correlate with the presence of errors. Manual inspections have further revealed a variety of errors, underscoring the need for a more nuanced evaluation framework, which is currently lacking. This resource paper addresses this gap by introducing a test collection for detecting and classifying errors in simplified texts. First, we propose a taxonomy of errors, with a formal focus on information distortion. Next, we introduce a parallel dataset of automatically simplified scientific texts. This dataset has been human-annotated with labels based on our proposed taxonomy. Finally, we analyze the quality of the dataset, and we study the performance of existing models to detect and classify errors from that taxonomy. These contributions give researchers the tools to better evaluate errors in ATS, develop more reliable models, and ultimately improve the quality of automatically simplified texts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。