提出新基准PersonalBench,量化大模型个性化生成与真人写作风格的差距。
PersonalBench: Measuring the Authorship Gap in LLM Personalization
- 用作者识别模型、LLM评判和风格统计三重方法评估个性化输出
- 模型生成文本与真人风格相似度仅0.484-0.508,低于人类间平均0.626
- 现有方法无法突破大模型自身风格主导,适合研究个性化边界者阅读
个性化文本生成旨在让大模型模仿特定个体的写作风格,但现有基准多关注任务准确率或偏好对齐,而非输出是否真正接近目标作者。本文提出PersonalBench,通过三种独立视角评估推理时个性化方法:训练好的作者识别模型(LUAR)、LLM作为评判者、以及自动化风格分析。在50位作者、1000次生成、两个模型族(Qwen 3, GLM-4)上发现,个性化方法确实生成了可区分作者的文本(LUAR AUC=0.918),但这种区分从未跨越人类与大模型之间的界限。所有方法在LUAR上与真实作者的相似度为0.484–0.508,低于人类间跨作者的最低分0.626(最高0.756)。大模型自身的作者指纹占据主导:生成文本与任意真人作者的距离,大于随机人类之间的距离。各方法在LUAR上差异极小(范围0.024),尽管在LLM评判中看似不同,该矛盾源于特征提取与风格建模之间的循环性。验证显示,LUAR在本语料库中可靠衡量作者身份(单篇AUC=0.76,多篇AUC=0.96)。PersonalBench作为校准工具发布,表明推理时个性化仅调节大模型风格,无法弥合与真人作者的鸿沟。
原文摘要 · Abstract (English)
Personalized text generation aims to make LLMs write in a specific individual's style, yet existing benchmarks measure task accuracy or preference alignment rather than whether the model's output actually resembles the target author's writing. We introduce PersonalBench, a benchmark that evaluates inference-time personalization methods through three independent lenses: LUAR (a trained authorship verification model), an LLM-as-judge, and automated stylometrics. Across 50 authors, 1,000 generations, and two model families (Qwen 3, GLM-4), we find that personalization methods do produce author-differentiated output (LUAR discriminates target authors within generated text at AUC=0.918) but this differentiation never crosses the human-LLM boundary. All methods achieve LUAR similarity to real authors in the range 0.484-0.508, below the cross-author human floor of 0.626 (ceiling 0.756). The LLM's own authorship fingerprint dominates: generated text is more distant from any human author than random humans are from each other. Methods are statistically indistinguishable from each other on LUAR (spread 0.024) despite appearing differentiated on the LLM judge, a discrepancy we trace to circularity between trait extraction and profile extraction. We validate that LUAR reliably measures authorship in our corpus (AUC=0.76 single-post, 0.96 multi-post). We release PersonalBench as a calibrated measuring stick: inference-time personalization modulates the LLM's style but does not bridge the gap to human authorship.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。