用反事实生成解决预测市场评论立场识别的不平衡问题
Stance Detection in Prediction Markets: Addressing Imbalanced Trader Commentary via Counterfactual Augmentation and Market Context
- 通过大模型生成反事实评论，增强少数类样本
- 加入市场上下文使反对立场召回率提升至0.45
- 50%合成数据效果最佳，过量反而降低性能
预测市场如Polymarket将群体信念转化为实时概率估计，其下方的交易者评论包含价格无法捕捉的方向性立场信号。本文首次针对预测市场评论开展立场识别研究，该领域具有极简表述、交易者特有语言和严重类别不平衡（仅8.7%评论反对市场结果）的特点。采用RoBERTa-base在4×3实验设计下微调：四种输入组合（2类/3类 × 是否含市场上下文）与三种增强条件（基线、50%合成、100%合成）。少数类样本通过Anthropic API驱动的LLM实现正向→反向的反事实转换生成。结果显示：（1）市场上下文是单一最显著因素，使三分类反对类召回率从0.10升至0.45；（2）反事实增强有条件有效，在弱配置中提升反对类F1（0.10→0.24），但在强配置中反而下降（2类+上下文宏F1由0.68降至0.50）；（3）50%增强为最优剂量，100%始终损害性能。基于注意力的可解释性分析为上述发现提供机制支持。
原文摘要 · Abstract (English)
Prediction markets such as Polymarket aggregate crowd beliefs into real-time probability estimates, and the comments traders post beneath each market contain rich directional stance signals that prices alone cannot capture. This work introduces the first stance detection study applied to prediction market commentary, a domain characterized by extreme brevity, trader- specific vernacular, and severe class imbalance (only 8.7% of comments oppose the market outcome). RoBERTa-base is fine-tuned across a 4 x 3 ablation: four input configurations ({2- class, 3-class} x {with/without market context}) and three augmentation conditions (baseline, 50% synthetic, 100% synthetic). Synthetic minority-class samples are generated via LLM-driven Pro -> Anti counterfactual flips using the Anthropic API. Results show that (1) market context is the single most impactful factor, raising 3-class Anti recall from 0.10 to 0.45; (2) counterfactual augmentation is conditionally effective, improving Anti F1 in weak configurations (0.10 -> 0.24) while degrading strong ones (2-class-ctx macro F1: 0.68 -> 0.50 at full dose); and (3) 50% augmentation is the optimal dose, with 100% consistently hurting performance. Attention-based interpretability analysis provides mechanistic support for all three findings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。