The standard paradigm of semantic segmentation is first training a multi-class classifier to estimate the probability distributions of the pixels’ possible labels and then employing a graphical model to encode contextual information to obtain a globally optimal solution. However, evaluating the quality of the trained classifier is a rarely investigated problem so far and its consequential influence on post-processing is also not yet analyzed. In this paper, we focus on the two problems and propose to estimate the confidence of the classifier by additionally learning a binary classifier via ensemble learning and convolutional neural network (CNN) respectively. Compared with the hand-crafted confidence measures, our learning-based methods are proved to be more effective. Moreover, the estimated confidences are employed to modulate the probability distributions to leverage the post-processing step and high-quality label maps can be established finally. We evaluate our methods on two public available datasets Sift Flow and Pascal-Context, and the results demonstrate that learning to predict confidences is an effective and promising strategy for improving semantic segmentation frameworks.