SIDInspector可提前发现推荐系统编码器的映射缺陷,避免训练后才发现问题。
SIDInspector: A Mapping-First Diagnostic Resource for Semantic-ID Tokenizers
- 以映射为中心设计诊断接口,独立于下游模型评估
- 发现网格容量提升使别名率从0.977降至0.000
- 适合构建推荐系统的开发者与模型调优者使用
语义标识(SID)分词器正被作为独立组件用于生成式推荐:导出的物品-编码映射成为后续序列生成器的地址空间。这些映射通常缺乏通用检查接口,导致覆盖盲区、全码别名、行为弱前缀、尾部压缩和前缀泛化等问题往往在下游训练后才被发现。我们提出SIDInspector,一种面向SID分词器的映射优先诊断工具。该工具定义了对物品映射、元数据、交互记录及可选生成轨迹的小型适配器合约,验证合约并报告映射级探针,包括利用率、别名性、邻域对齐度、流行度分配与结构成本,并支持时间演化与生成轨迹钩子。SIDInspector在不依赖下游排行榜得分的情况下表征导出映射。当前版本涵盖四种分词器线:两种容量设置下的GRID风格RQ-KMeans,以及同物品音乐协议下的受限ReSID/GAOQ导出,另有LETTER与LC-Rec物品索引。在该协议中,添加GRID容量使D2别名率从0.977降至0.779,而ReSID/GAOQ下为0.000;D3始终低于确定性类别前缀控制(0.055–0.154 对比 0.447)。该工具将容量、可寻址性与行为前缀对齐视为独立的制品属性。额外领域检验也显示相同诊断区分。受限固定重排序探针将前缀对齐与候选曝光关联,而最终排名质量仍为下游模型问题。
原文摘要 · Abstract (English)
Semantic-ID (SID) tokenizers are increasingly reused as standalone artifacts in generative recommendation: an exported item-to-code mapping becomes the address space that a later sequence generator must use. These mappings rarely come with a common inspection interface, so coverage gaps, full-code aliasing, behaviorally weak prefixes, tail compression, and prefix fan-out are often found only after downstream training. We present SIDInspector, a mapping-first diagnostic resource for SID tokenizer artifacts. SIDInspector defines a small adapter contract over item mappings, metadata, interactions, and optional generator traces; validates the contract; and reports mapping-level probes for utilization, aliasing, neighborhood alignment, popularity allocation, and structural cost, with hooks for temporal churn and generator traces. SIDInspector characterizes exported mappings independently of downstream leaderboard scores. The release covers four tokenizer lines: GRID-style RQ-KMeans under two capacity settings and a bounded ReSID/GAOQ export in a same-item Musical protocol, plus released LETTER and LC-Rec item-index artifacts. In that protocol, D2 aliasing falls from 0.977 to 0.779 under added GRID-style capacity and to 0.000 for ReSID/GAOQ, while D3 remains below a deterministic category-prefix control (0.055-0.154 versus 0.447). SIDInspector exposes capacity, addressability, and behavioral prefix alignment as distinct artifact properties before method-level evaluation. Additional domain checks show the same diagnostic distinction. Bounded fixed-reranker probes connect prefix alignment to candidate exposure, while final ranking quality remains a downstream model question.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。