ML0095 Confidence Interval

A confidence interval is a range computed from sample data that is designed to contain an unknown population parameter with a specified probability under repeated sampling. The correct frequentist interpretation of a 95% confidence interval is: if you repeated the experiment many times, computing a new interval from each sample, about 95% of those intervals would contain the true parameter. It is a statement about the procedure, not about any single interval. Once data is observed, the true parameter is either inside your specific interval or not (probability 0 or 1); you cannot say “there is a 95% probability the parameter is in this interval,” because that is a Bayesian credible interval interpretation requiring a prior. Sequential A/B testing platforms use “anytime-valid” confidence sequences that maintain this coverage guarantee across continuous monitoring, and asymmetric experiment designs treat confidence interval width as a design parameter controlled through sample allocation between test and control groups.

(1) Procedure vs Single Interval: the 95% refers to the long-run coverage rate of the interval-construction procedure across hypothetical repeated samples, not the probability that the true parameter is in any one observed interval.
(2) Frequentist vs Bayesian: saying “95% probability the parameter is in this interval” is a Bayesian credible interval statement; a 2024 paper showed this distinction has real business impact, as frequentist estimators overstate A/B test effect sizes (winner’s curse) while Bayesian estimators with correct priors do not.
(3) Production Use: anytime-valid confidence sequences enable canary testing so the 95% coverage holds throughout continuous monitoring, not just at a pre-specified endpoint; asymmetric experiment designs optimize CI width by nonlinearly increasing control group size when shrinking test group size (40% test reduction requires 3x control increase).

Figure 1: The frequentist interpretation: across 20 repeated experiments, about 19 of the 95% confidence intervals contain the true parameter (vertical line); the 95% is a property of the procedure, not of any single interval.

The width of a confidence interval depends on three things: the confidence level (higher confidence means wider intervals), the sample size (larger n means narrower intervals), and the variability of the data (higher variance means wider intervals). For a mean with known variance, the 95% CI is the sample mean plus or minus 1.96 standard errors. When the variance is unknown and estimated from the sample, the t-distribution replaces the normal, giving slightly wider intervals for small samples. A critical production issue is peeking: repeatedly checking whether a confidence interval excludes zero and stopping when it does inflates the false positive rate, because each check is a separate test. Anytime-valid confidence sequences solve this by using a test statistic whose coverage guarantee holds uniformly over time, so you can monitor continuously without breaking the 95% promise. Conformal prediction (2024-2025) extends the idea to distribution-free prediction intervals for ML models, providing finite-sample coverage without normality assumptions.

Mathematical Formulation:

Where:

• is the sample mean, is the population standard deviation (replaced by sample standard deviation with the t-distribution when unknown), and is the sample size.
• is the critical value of the standard normal at level ; for 95% confidence, and .
• The probability statement is over the random interval (which depends on ), not over the fixed parameter ; this is the frequentist distinction. The interval either covers or not for any single sample, but the procedure covers it of the time across repeated samples.