解决调查数据中结构跳过和有序变量的缺失值填补问题
TabSODA: Tabular Diffusion based Imputation with Skip Pattern Detection and Ordinal Awareness
- 基于扩散模型,通过损失函数传递结构跳过信息
- 对有序变量使用累积概率标量编码,提升填补精度23.7%
- 适合处理含跳过逻辑的大型调查数据,如烟草与药物使用调查
大规模调查中的缺失数据填补面临两大挑战:一是问卷设计导致的结构性跳过(如某题不适用),常被误当作缺失值;二是有序响应(如“低、中、高”)被错误处理为无序类别。本文提出TabSODA(表格式扩散填补带跳过检测与有序感知),基于阐明式扩散模型(EDM)框架,采用期望最大化(EM)策略。它通过去噪损失和逆时序采样传播结构跳过信号,并对有序变量使用累积概率标量潜变量,同时保留名义变量的模拟比特编码。当存在代码本跳过掩码时直接使用;否则,通过基于CART的跳过模式挖掘器从原始响应和问卷顺序估计掩码(TabSODA+SKIP)。在PATH与NSDUH两项美国全国代表性调查中,相比最强基线,TabSODA在MCAR、MAR和MNAR缺失场景下,有序变量的综合误差(MACE)降低最高达23.7%,分类准确率提升最高9%。跳过挖掘器在两数据集上达到接近完美的精确率,使TabSODA+SKIP性能逼近代码本掩码版本。
原文摘要 · Abstract (English)
Missing data imputation in large-scale surveys faces two challenges that are not well handled by current tabular diffusion methods. First, \emph{structural skips}, cells made inapplicable by questionnaire design, should not be imputed but are often conflated with item nonresponse. Second, \emph{ordinal} responses encode ordered categories, yet most pipelines treat them as nominal levels through one-hot or analog-bit encodings. We introduce \textbf{TabSODA} (\textbf{Tab}ular diffusion with \textbf{S}kip pattern detection and \textbf{O}r\textbf{d}inal \textbf{A}wareness), an Expectation-Maximization (EM)-based diffusion imputer built on the Elucidated Diffusion Model (EDM) framework. TabSODA propagates structural skips through the denoising loss and reverse-time sampler, and represents ordinal variables with cumulative-probit scalar latents while retaining analog-bit encodings for nominal variables. When a codebook skip mask is available, TabSODA uses it directly; otherwise, the TabSODA+SKIP variant estimates the mask from raw responses and questionnaire order using a CART-based skip-pattern miner. On Population Assessment of Tobacco and Health (PATH) study and the National Survey on Drug Use and Health (NSDUH), two nationally representative U.S.\ surveys, TabSODA reduces ordinal MACE by up to $23.7\%$ and improves categorical accuracy by up to $9\%$ over the strongest baseline across MCAR, MAR, and MNAR masking. The skip miner achieves near-perfect precision on both datasets, allowing TabSODA+SKIP to closely track the codebook-mask variant.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。