arXiv · Jul 29, 2026
Cost-Sensitive Conformal Prediction and Human-in-the-Loop Abstention for Imbalanced High-Stakes Decision Support: A Multi-Domain Benchmark
A benchmark study across 15 real-world imbalanced tabular datasets, 7 classification models, 3 calibration techniques, and 10 random seeds (3,150 runs) found that standard marginal conformal prediction severely under-covers rare minority classes (as low as 0.5% coverage), while Mondrian (class-conditional) conformal prediction restores valid minority-class coverage with an average improvement of 61.7 percentage points (p < 1e-80). Combining Mondrian CP with cost-controlled abstention further reduces expected decision cost compared to standard decision boundaries, confidence-based rejectors, and risk-controlled rejectors.
What happened
High-stakes decision systems in credit scoring, fraud detection, healthcare, and industrial safety require reliable uncertainty quantification under severe class imbalance and asymmetric error costs. Standard marginal conformal prediction (CP) provides valid overall coverage guarantees; however, we show that it severely under-covers rare, costly minority classes, with minority-class coverage dropping to as low as 0.5% on certain datasets. To characterize and address this limitation, we conduct a comprehensive benchmark comparing marginal CP, class-conditional (Mondrian) CP, and cost-controlled abstention mechanisms across 15 real-world imbalanced tabular datasets, 7 classification models, 3 probability calibration techniques, and 10 random seeds, resulting in 3,150 experimental runs. Our results show that Mondrian CP restores valid minority-class coverage, achieving an average minority-coverage improvement of 61.7 percentage points over marginal CP (p < 1e-80). Furthermore, combining Mondrian CP with cost-controlled abstention significantly reduces expected decision cost compared with standard decision boundaries, confidence-based rejectors, and risk-controlled rejectors under realistic cost structures.
Technical significance
Mondrian conformal prediction provides class-conditional coverage guarantees by partitioning calibration data by class, ensuring valid coverage for each class independently. This addresses the failure mode of marginal CP, which can achieve overall coverage by over-covering the majority class while under-covering the minority class. The integration of cost-controlled abstention allows the system to defer decisions to a human when the expected cost of an automated decision exceeds the cost of human intervention, optimizing the trade-off between accuracy and cost in imbalanced settings.
Industry impact
The findings are directly applicable to industries where rare but costly events are critical, such as fraud detection in finance, disease diagnosis in healthcare, and fault detection in industrial systems. The demonstrated failure of standard conformal prediction in these settings highlights a significant risk in deploying such methods without class-conditional adjustments. The proposed combination of Mondrian CP and cost-controlled abstention offers a practical framework for building more reliable and cost-effective decision support systems.
What to watch
Next signals to watch include adoption of Mondrian CP in production ML pipelines for high-stakes applications, development of open-source libraries implementing cost-controlled abstention with conformal prediction, and further research extending these methods to non-tabular data (e.g., images, text) and online learning settings. Regulatory bodies may also begin to require class-conditional coverage guarantees for AI systems in sensitive domains.
Decision value
Organizations deploying AI for high-stakes decisions can reduce expected costs by combining Mondrian conformal prediction with cost-controlled abstention, as it ensures reliable uncertainty estimates for rare but costly events and intelligently defers to human experts when automated decisions are too risky. This can lead to fewer costly errors, improved trust in AI systems, and better compliance with fairness and safety standards.