arXiv:2511.20409cs.CL2025-11

提出统一评估文本归一化质量的多指标框架,避免误判语义损失。

NormEval: A Unified Multi-Metric Framework for Evaluating Semantic Fidelity in Text Normalization

  • 设计五项互补指标,从效率、效用、形态保真三维度评估
  • 实验证明任一指标缺失都会导致评估误导,所有组件不可或缺
  • 特别引入字符级距离检测恶意变异,适合医疗法律等高风险场景

文本归一化方法如词干提取和词形还原是自然语言处理流程的基础。随着针对多种语言的新工具不断出现,评估方法仍碎片化,仅依赖压缩比、下游准确率或序列到序列预测分数,无法区分有益的词汇缩减与有害的语义失真。此外,文本归一化支撑临床决策支持和法律文档分析等高风险智能系统,需要严谨的评估方法。本文提出NormEval,一个统一的多语言评估框架,包含五个互补指标:压缩比(CR)、模型性能差值(MPD)、信息保留分数(IRS)、算法有效性分数(AES)和平均归一化编辑距离(ANLD)。这些指标从宏观效率、下游实用性及微观形态保真三个维度评估归一化质量。框架实现安全门假设:ANLD作为内在结构健康检查,利用字符级差异(Δ)揭示宏观嵌入和下游任务所掩盖的激进变异。在孟加拉语和英语数据集上的全面消融实验表明,所有组件均不可或缺,任一指标缺失均导致至少一项评估维度下降,最终造成算法排名误导。

原文摘要 · Abstract (English)

Text normalization methods such as stemming and lemmatization are fundamental components of NLP pipelines. As new normalization tools are developed for diverse languages, evaluation methodologies remain fragmented, relying on Compression Ratio, downstream accuracy, or sequence-to-sequence prediction scores in isolation, failing to distinguish between beneficial vocabulary reduction and harmful semantic distortion. Moreover, text normalization underpins intelligent systems in high-stakes domains, including clinical decision support and legal document analysis, and principled evaluation methodology is essential. This paper proposes NormEval, a unified, multilingual evaluation framework comprising five complementary metrics: Compression Ratio (CR), Model Performance Delta (MPD), Information Retention Score (IRS), Algorithm Effectiveness Score (AES), and Average Normalized Levenshtein Distance (ANLD). These metrics assess normalization quality across three dimensions: macro-level efficiency, downstream utility, and micro-level morphological fidelity. The framework operationalizes a Safety Gate hypothesis: ANLD functions as an intrinsic structural hygiene check, utilizing character-level divergence ($Δ$) to reveal aggressive mutations that macro-level embeddings and downstream tasks mask. Comprehensive ablation experiments on both Bangla and English datasets show that all the components are indispensable, and that the removal of any individual metric leads to a decrease in at least one evaluation aspect, which ultimately results in misleading algorithm rankings.

文本归一化评估框架语义保真

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。