发现语义标识生成推荐评估存在严重偏差，提出纠正方法。
Faithful Evaluation of Semantic-ID Tokenizers for Generative Recommendation
- 引入碰撞修正评估法，基于生成序列计算更准确的指标
- 实测碰撞率高达30.52%，导致命中率虚高超100%
- 适用于需可靠对比推荐编码器的研究者
基于语义标识（SID）的生成式推荐将每个物品表示为离散的SID序列，传统评估方式通过比对生成序列与目标序列进行判断。该方法仅在每个SID序列唯一对应一个物品时才准确。然而实际中存在大量SID冲突：在多个评估的SID分词器与数据集上，碰撞率高达30.52%，导致SID级Hit@10相比物品级虚高最多达103.36%。为此，我们提出碰撞修正评估（CCE），定义基于碰撞感知的物品级指标（ItemHit@K, ItemNDCG@K），并设计零碰撞重分配（ZCR），通过最小成本重分配构建无冲突的SID映射。在四个数据集和五个代表性分词器上应用后发现，评估膨胀程度随碰撞率上升，足以在物品级重评中反转分词器间的比较结果。这质疑了以往工作中的SID级排序可靠性，表明忠实评估需采用物品级校正或零碰撞分配。
原文摘要 · Abstract (English)
Generative recommendation based on Semantic IDs (SIDs) represents each item as a discrete sequence of SIDs and is conventionally evaluated by matching the generated SID sequence against the target item's SID sequence. This evaluation is faithful only when each SID sequence uniquely identifies one item. In practice, SID collisions violate this condition: across the evaluated SID tokenizers and datasets, collision rates reach 30.52%, and SID-level Hit@10 is inflated by up to 103.36% relative to item-level Hit@10. To address this evaluation gap, we introduce Collision-Corrected Evaluation (CCE), which defines collision-aware item-level metrics (ItemHit@K, ItemNDCG@K) computed from generated SID sequences, and Zero-Collision Reassignment (ZCR), which constructs zero-collision SID assignments for existing tokenizers via minimum-cost reassignment. Applying these methods to four datasets and five representative SID tokenizers, we find that metric inflation scales with collision rate and is large enough to flip pairwise tokenizer comparisons under item-level re-evaluation. This finding calls into question the reliability of SID-level rankings reported in prior work and indicates that faithful tokenizer evaluation requires item-level correction or zero-collision SID assignments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。