How Curriculum Learning Impacts Model Calibration

Despite the significant progress made on deep learning models, concerns yet exist when a trained model is deployed to real-world applications. Model calibration is a key consideration that has recently attracted more attention---a learned model should not only achieve high predictive performance but also attain that with a proper level of confidence---a mismatch between predictive performance and confidence creates miscalibration and hence raises concerns about trusting a (miscalibrated) model. Even with the importance of the problem and many recent research efforts, calibration has not been fully understood yet, particularly when it faces the common challenges that deep learning models struggle with: specifically limited training resources and noisy data. In this paper, we study calibration emphasizing these scenarios. We particularly investigate the effect of curriculum learning, which, inspired by human curricula, leverages a guided learning regime to improve model generalization and has been found to improve predictive performance in the aforementioned cases. Specifically, we provide an empirical understanding on the impact of curriculum learning on model calibration under a variety of general contexts. Our studies suggest the following: most of the time curriculum learning has a negligible effect on calibration, but in certain cases under the context of limited training time and noisy data, curriculum learning can substantially reduce calibration error in a manner that cannot be explained by dynamically sampling the dataset. Second, curriculum and anti-curriculum learning appear to have nearly identical effects on model calibration. Lastly, the choice of pacing function and its parameters in curriculum learning can significantly impact model calibration, indicating that extra care should be taken to minimize the risk of severe model miscalibration. We hope the empirical insights will help us better understand calibration and guide the utilization of curriculum learning in practice.