Calibrating User Response Predictions in Online Advertising

Predicting user response probability such as click-through rate (CTR) and conversion rate (CVR) accurately is essential to online advertising systems. To obtain accurate probability, calibration is usually used to transform predicted probabilities to posterior probabilities. Due to the sparsity and latency of the user response behaviors such as clicks and conversions, traditional calibration methods may not work well in real-world online advertising systems. In this paper, we present a comprehensive calibration solution for online advertising. More specifically, we propose a calibration algorithm to exploit implicit properties of predicted probabilities to reduce negative impacts of the data sparsity problem. To deal with the latency problem in calibrating delayed responses, e.g., conversions, we propose an estimation model to leverage post-click information to approximate the real delayed user responses. We also notice that existing metrics are insufficient to evaluate the calibration performance. Therefore, we present new metrics to measure the calibration performance. Experimental evaluations on both real-world datasets and online advertising systems show that our proposed solution outperforms existing calibration methods and brings significant business values.