Online Control of the False Coverage Rate and False Sign Rate

The reproducibility debate has caused a renewed interest in changing how one reports uncertainty, from $p$-value for testing a null hypothesis to a confidence interval (CI) for the corresponding parameter. When CIs for multiple selected parameters are being reported, the analog of the false discovery rate (FDR) is the false coverage rate (FCR), which is the expected ratio of number of reported CIs failing to cover their respective parameters to the total number of reported CIs. Here, we consider the general problem of FCR control in the online setting, where there is an infinite sequence of fixed unknown parameters ordered by time. While much progress has been made in online testing, a procedure controlling the FDR does not automatically translate to a (nontrivial) procedure that controls the FCR. Therefore, the problem of online FCR control needs to be treated separately. We propose a novel solution to the problem which only requires the scientist to be able to construct a marginal CI at any given level. If so desired, our framework also yields online FDR control as a special case, or even online sign-classification procedures that control the false sign rate (FSR). Last, all of our methodology applies equally well to prediction intervals, having particular implications for selective conformal inference.