Applications of Class-Conditional Conformal Predictor in Multi-Class Classification

Abstract

In many prediction problems, it is beneficial to obtain confidence estimates for the classification output. We consider the problem of estimating confidence sets in multiclass classification of real life datasets. Building on the theory of conformal predictors, we derive a class-conditional conformal predictor. This allows us to calibrate the confidence estimates in a class specific fashion, resulting in a more precise control of the prediction error rate for each class. We show that the class-conditional conformal predictor is asymptotically valid, and demonstrate that it indeed provides better calibration and efficiency on benchmark digit recognition datasets. In addition, we apply the cla..