用控制令牌微调开源模型,实现可调节的文本简化。
Taming CATS: Controllable Automatic Text Simplification through Instruction Fine-Tuning with Control Tokens
- 通过离散控制令牌指导模型生成不同可读性与压缩率的文本。
- 1-3B小模型表现媲美大模型,但压缩控制效果受限于数据信号不足。
- 提出基于误差的评估方法,解决传统指标无法衡量控制力的问题。
可控自动文本简化(CATS)能生成符合用户需求的文本,但可控性常被当作解码问题处理,且评估指标未能真实反映控制能力。我们发现ATS中的可控性受限于数据和评估方式。为此,提出一种无需领域依赖的CATS框架,基于指令微调与离散控制令牌,引导开源模型达到目标可读性水平和压缩率。在三种模型家族(Llama、Mistral、Qwen;1-14B)和四个领域(医学、公共管理、新闻、百科)上测试发现,1-3B小模型表现可与大模型媲美,但可靠可控性强烈依赖训练数据中目标属性的充分变化。可读性控制(FKGL、ARI、Dale-Chall)学习稳定,而压缩控制表现不佳,因现有语料中信号变异不足。我们进一步证明标准简化与相似性指标不足以衡量控制效果,需采用基于误差的目标-输出对齐度量。最后,采样与分层实验表明,简单划分可能引入分布偏差,损害训练与评估效果。
原文摘要 · Abstract (English)
Controllable Automatic Text Simplification (CATS) produces user-tailored outputs, yet controllability is often treated as a decoding problem and evaluated with metrics that are not reflective to the measure of control. We observe that controllability in ATS is significantly constrained by data and evaluation. To this end, we introduce a domain-agnostic CATS framework based on instruction fine-tuning with discrete control tokens, steering open-source models to target readability levels and compression rates. Across three model families with different model sizes (Llama, Mistral, Qwen; 1-14B) and four domains (medicine, public administration, news, encyclopedic text), we find that smaller models (1-3B) can be competitive, but reliable controllability strongly depends on whether the training data encodes sufficient variation in the target attribute. Readability control (FKGL, ARI, Dale-Chall) is learned consistently, whereas compression control underperforms due to limited signal variability in the existing corpora. We further show that standard simplification and similarity metrics are insufficient for measuring control, motivating error-based measures for target-output alignment. Finally, our sampling and stratification experiments demonstrate that naive splits can introduce distributional mismatch that undermines both training and evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。