用压缩算法衡量AI文本修改难度，更贴近真实编辑耗时。
Assessing Human Editing Effort on LLM-Generated Texts via Compression-Based Edit Distance
- 基于LZ77压缩原理，量化原文与修改后文本的信息差异。
- 在真实编辑数据上，与实际编辑时间高度相关（相关系数0.82）。
- 适合评估复杂编辑操作，尤其适用于人机协作场景研究。
评估大语言模型生成文本所需的人工编辑量，对理解人机交互、提升自动文本生成质量至关重要。现有编辑距离度量方法（如Levenshtein、BLEU、ROUGE、TER）在应对块级修改等大幅改动时往往失准。本文提出一种基于Lempel-Ziv-77算法的新型压缩编辑距离度量，利用文本压缩特性衡量原始文本与修改后文本之间的信息差异。在真实人类编辑数据集上的实验表明，该方法与实际编辑时间和努力高度相关（相关系数0.82）。我们还发现，大语言模型隐含地具备与该度量一致的编辑速度感知能力。相比现有方法，本方法在捕捉复杂编辑操作方面更具优势，且计算效率为线性。代码与数据已公开于：https://github.com/NDV-tiime/CompressionDistance。
原文摘要 · Abstract (English)
Assessing the extent of human edits on texts generated by Large Language Models (LLMs) is crucial to understanding the human-AI interactions and improving the quality of automated text generation systems. Existing edit distance metrics, such as Levenshtein, BLEU, ROUGE, and TER, often fail to accurately measure the effort required for post-editing, especially when edits involve substantial modifications, such as block operations. In this paper, we introduce a novel compression-based edit distance metric grounded in the Lempel-Ziv-77 algorithm, designed to quantify the amount of post-editing applied to LLM-generated texts. Our method leverages the properties of text compression to measure the informational difference between the original and edited texts. Through experiments on real-world human edits datasets, we demonstrate that our proposed metric is highly correlated with actual edit time and effort. We also show that LLMs exhibit an implicit understanding of editing speed, that aligns well with our metric. Furthermore, we compare our metric with existing ones, highlighting its advantages in capturing complex edits with linear computational efficiency. Our code and data are available at: https://github.com/NDV-tiime/CompressionDistance
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。