The “95%” in a 95% confidence interval is the hit rate of the procedure when it is repeated. Whether the one interval in front of you contains the population mean is unknown. And depending on the shape of the data, that hit rate may fall short of 95%.
Stating the population mean as a range
As on the previous page, the population is the 265 primate species with recorded body mass. The population mean is 5,881 grams. In a real study the whole population cannot be observed; all you have is the species you selected, that is, a sample.
Stating the sample mean itself as “the population mean is this value” is point estimation. But the sample mean changes every time you select again. Saying instead “the population mean is probably within this range” is interval estimation, and the range is called a confidence interval.
- Point estimation
- Stating a population value as a single number calculated from a sample.
- Interval estimation
- Stating a population value as a range with a lower and an upper end.
- Confidence interval
- The range produced by interval estimation. A 95% confidence interval is built so that the method hits 95% of the time.
- Coverage
- The proportion of intervals that contain the population mean when the procedure is repeated many times.
Building a single interval
Here is a sample of 15 species selected from the 265. The mean of these 15 species is 6,621 grams, and the 95% confidence interval is 1,052 to 12,190 grams. This interval contains the population mean of 5,881 grams.
The dots above are the body masses of the 15 selected species; the horizontal line below is the 95% confidence interval. The rightmost dot is the Sumatran orangutan (Pongo abelii, 39,696 grams), and this one species pulls the sample mean up considerably.
So is the probability that this interval contains the population mean 95%? The answer is no. The population mean is a fixed value, 5,881 grams, and once calculated the interval is a fixed range too. A fixed value either lies within a fixed range or it does not. Probability applies to the procedure: selecting a new sample and building the interval again.
- Intervals built
- —
- Contained the mean
- —
- Proportion (coverage)
- —
- Interval width (median)
- —
The chart shows the most recent 100 intervals, newest at the bottom. Species are selected with replacement: each one is put back before the next is chosen. Because random numbers are used, the proportion changes slightly with every click.
How it works: building the interval, and why 95% is not kept
The intervals on this page are built with the following formula.
The sample standard deviation is the spread of the values within the sample. Dividing it by √n gives the standard error, an estimate of how much the sample mean fluctuates. t is a multiplier that adds the extra uncertainty from estimating the standard deviation from the sample; the smaller n is, the larger it gets (2.78 for n=5, 2.14 for n=15, 2.02 for n=40, 1.98 for n=100).
The formula assumes that the distribution of the sample mean is close to bell-shaped. With a strongly right-skewed population such as body mass, this assumption breaks down at small n. Only 7 of the 265 species weigh 20,000 grams or more, and most samples contain none of them. Both the sample mean and the sample standard deviation then come out small, and the whole interval shifts below the population mean. In the simulation, the sample mean fell below the population mean in 67.5% of samples at n=15.
The table below shows the actual coverage, measured by building 10,000 intervals for each n.
| Species selected, n | t | Raw body mass | Log body mass |
|---|---|---|---|
| 5 species | 2.78 | 83.9 % | 92.6 % |
| 15 species | 2.14 | 73.4 % | 94.7 % |
| 40 species | 2.02 | 74.3 % | 94.6 % |
| 100 species | 1.98 | 82.6 % | 95.3 % |
Results of selecting n species from the 265 with replacement and building a 95% confidence interval, repeated 10,000 times each (fixed random seed). For raw body mass, the proportion containing the population mean of 5,881 g; for log body mass, the proportion containing the population geometric mean of 2,120 g.
For raw body mass, the actual hit rate stays in the 70s and low 80s of percent, even though the interval is called “95%”. Nearly all of the intervals that missed did so on the low side of the population mean. Taking the log of body mass makes the distribution nearly symmetric, and coverage returns almost to its design value, 94.7% already at n=15. Note, however, that an interval built on the log scale and converted back is an interval for the geometric mean, not the arithmetic mean. Which to build depends on what you want to estimate.
Common misconceptions
“There is a 95% probability that the population mean lies in this interval”
The population mean is a fixed value, and a calculated interval is a fixed range. The mean is either in it or not. The 95% is a property of the procedure: the proportion of intervals that contain the population mean when you keep rebuilding them the same way.
“If it says 95% confidence interval, the hit rate is 95%”
It is 95% only when the assumptions of the formula hold. As the table shows, for the strongly skewed raw body mass values it was 73.4% with 15 species and still only 82.6% with 100 species. Check the shape of the data and how the interval was built, not the name of the interval.
“The narrower the interval, the more accurate the hit”
Narrowness is precision, not correctness. With raw body mass and n=15, over 10,000 intervals the median width of those that missed the population mean was 3,539 g, narrower than the 5,929 g of those that hit. None of the samples behind the missed intervals included any of the 7 species weighing 20,000 grams or more. Samples that miss the large species underestimate the spread and produce narrow, overconfident intervals.
“Increase n, and the hit rate quickly approaches 95%”
Increasing n makes intervals narrower, but how far the hit rate recovers depends on the shape of the population. For raw body mass it was 74.3% even at n=40 and 82.6% at n=100. On the log scale it reaches 94.7% at n=15.
Covered in the courses
The course sites are in Japanese and open to anyone. To go further, visit the page for each session.
Data: Jones, K. E. et al. (2009) PanTHERIA: a species-level database of life history, ecology, and geography of extant and recently extinct mammals. Ecology 90: 2648. The variable “body mass” was used, and the 265 of 376 primate species with recorded body mass were taken as the population.
Coverage in the table and the figures in the misconceptions section come from a simulation that draws samples from these 265 species with replacement (10,000 repetitions each, fixed random seed). t is the 97.5% point of the t distribution with n−1 degrees of freedom.