用规则+词袋方法自动分类零售商品名，提升价格比较的准确性。
Machine Learning for Coding Retail Product Names to Consumer-Price Categories: A Rule-plus-Bag-of-Words Pipeline with Reliability-Weighted Human-in-the-Loop Labeling
- 先用规则匹配关键词，再用二分类模型确认类别，流程可复现。
- 字符n-gram逻辑回归在六类中平均F1达0.997，词序特征无效。
- 少量人工标注（每类约66条）结合动态权重可有效提升标注质量。
消费者价格测量越来越多依赖扫描、网络抓取和交易/收据数据等替代数据源，这些数据中的商品描述短且嘈杂，无标准编码，需先映射到消费分类（如联合国COICOP体系）才能进行价格比较。本文研究该映射任务作为通用可复现方法：(i) 对噪声商品名进行文本标准化与分词；(ii) 基于前缀树（trie）的规则预分类器，利用各分类的关键词与停用词；(iii) 每类独立的二分类确认模型。为实现大规模标注，采用人机协同协议：标注员给出二元有效/拒绝判断，并通过动态更新的可靠性权重聚合；模型同步参与，实现持续微调。在一个可复现的合成基准上（六类类似COICOP类别），在统一评估协议下，轻量级模型表现最优，顺序敏感模型无帮助：字符n-gram逻辑回归在所有类别中均领先（平均F1=0.997），词序特征未提升性能，小型CNN/LSTM在小数据场景中最差。仅靠trie规则可处理32%-50%的商品，因此学习阶段必不可少，每类约66个标注样本即足够。蒙特卡洛分析表明，可靠性加权投票仅略优于简单多数投票，而Dawid-Skene方法显著更优。所有代码与合成数据均已开源。
原文摘要 · Abstract (English)
Consumer-price measurement increasingly draws on alternative data sources -- scanner, web-scraped, and transaction/receipt data -- whose product descriptions are short, noisy, and carry no standard product code, so each item must first be mapped to a consumption classification (e.g., the UN COICOP scheme) before prices can be compared. This paper studies that mapping as a general, reproducible method. The pipeline is: (i) text normalization and tokenization of noisy item names; (ii) a prefix-tree (trie) rule-based pre-classifier driven by per-category key-phrases and stop-phrases; and (iii) a per-category binary confirmation model. For labels at scale we use a human-in-the-loop protocol in which annotators give a binary valid/reject judgment aggregated by a dynamically updated reliability weight; the model joins the same rule, enabling continual fine-tuning. On a reproducible synthetic benchmark of six COICOP-like categories, under one matched protocol, cheap models win and order-sensitive ones do not help: a character n-gram logistic regression tops every category (mean F1 = 0.997), word-order features add nothing, and small CNN/LSTM models are the weakest in this small-data regime. The trie alone admits only 32-50% of items, so the learned stage is necessary, and about 66 labels per category suffice. A Monte-Carlo study of the labeling protocol is self-critical: the reliability-weighted vote barely beats plain majority while Dawid-Skene recovers labels markedly better. No proprietary or production data are used; all code and synthetic data are released at https://doi.org/10.5281/zenodo.20909563
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。