A hypothesis test assumes that there is really no difference and counts how often a difference like the one you see would arise. That proportion is the p-value, and it is not the size of the difference.
Do different families really differ in body mass?
Primate body mass seems to follow different patterns in different families. But when one family looks heavier than another, you cannot tell just by looking whether that reflects a real difference between families or whether the family simply happened to include more heavy species.
Here we compare three pairs of families among the species with a recorded body mass. Body mass varies several-fold even within a family, so we average the logarithms and express the difference as “how many times heavier.” This average is called the geometric mean.
| Families compared | Species | Geometric mean | Heavier by | p-value |
|---|---|---|---|---|
| Lemuridae and Indriidae | 15 and 8 | 2,140 g and 3,398 g | 1.59× | about 0.08 |
| Galagidae and Lorisidae | 13 and 8 | 286 g and 443 g | 1.55× | about 0.25 |
| Cebidae and Pitheciidae | 39 and 22 | 590 g and 1,493 g | 2.53× | below 0.0001 |
Each dot is one species. Vertical lines mark each family's geometric mean. The p-values come from a two-sided permutation test that shuffled the family labels at random 20,000 times.
The first and second pairs both differ by about 1.6 times. Yet their p-values differ roughly threefold, and neither falls below the conventional threshold of 0.05. Only the third pair gives a result that is clearly hard to attribute to chance. The hands-on panel below shows where this difference comes from.
- Observed difference
- —
- Shuffles
- —
- At least as large
- —
- p-value (that share)
- —
A shuffle pools the species of both families and then reassigns them at random, keeping the original number of species in each. It reproduces by hand the assumption that family and body mass are unrelated.
In the chart below, red bars are differences as large as or more extreme than the observed one. This is a two-sided test, so it counts differences in either direction.
How it works: the null hypothesis and the p-value
A test begins with a null hypothesis. Here it is: “family has nothing to do with body mass, and the observed difference arose by chance from which species happened to fall in which family.”
If the null hypothesis is true, shuffling the family labels should not change how differences turn up. So we shuffle many times and build the distribution of differences produced by chance alone. The proportion of that distribution lying at least as far out as the observed difference is the p-value.
The smaller the p-value, the harder it is to explain the difference by chance alone. By convention a p-value below 0.05 is called “statistically significant,” but this threshold is an agreement among researchers. Nothing in nature switches at 0.05.
The p-value depends both on the size of the difference and on the spread and number of the data. In Lemuridae and Indriidae, the eight Indriidae species range widely, from 829 g to 8,565 g. With few species and a large spread, shuffling alone often produces large differences. That is why a 1.59-fold difference still cannot be clearly distinguished from chance.
- Null hypothesis
- The hypothesis that there is “no difference” or “no relationship.” The test does its calculation assuming it is true.
- Permutation test
- A test that shuffles group labels at random to build the distribution of differences produced by chance alone.
- p-value
- The proportion of differences at least as extreme as the observed one, if the null hypothesis is true.
- Significance level
- The threshold for calling a result “significant.” By convention, 0.05.
Common misconceptions
“The p-value is the probability that the null hypothesis is true”
It is not. The p-value is the proportion of differences at least as large as the observed one, assuming the null hypothesis is true. For Lemuridae and Indriidae, p ≈ 0.08 does not mean “there is an 8% chance that family and body mass are unrelated.”
“If it is not significant, there is no difference”
Not significant means “could not be distinguished from chance.” Indriidae are 1.59 times heavier than Lemuridae in geometric mean, yet a comparison of 8 and 15 species stopped at p ≈ 0.08. That does not show there is no difference.
“The smaller the p-value, the larger the difference”
Galagidae and Lorisidae differ by 1.55 times, and Lemuridae and Indriidae by 1.59 times, almost the same. Yet their p-values are about 0.25 and about 0.08, roughly three times apart. Whenever you look at a p-value, look at the effect size, the fold difference, alongside it.
“Compare many pairs and report the ones that come out significant”
Comparing every pair among the 14 families with records for at least two species means 91 tests. Even if no family differed at all, testing at a significance level of 0.05 would, on average, make 4 to 5 pairs “significant” by chance.
Covered in the courses
The course sites are in Japanese and open to anyone. To go further, visit the page for each session.
Data: Jones, K. E. et al. (2009) PanTHERIA: a species-level database of life history, ecology, and geography of extant and recently extinct mammals. Ecology 90: 2648. Variable: adult body mass (g). Of the 265 species with a recorded body mass, the 105 species in the six families compared were used (Lemuridae 15, Indriidae 8, Galagidae 13, Lorisidae 8, Cebidae 39, Pitheciidae 22). Family assignments follow the teaching materials for the course Statistics I.
The p-values in the table come from 20,000 permutations. The p-value in the hands-on panel uses fresh random numbers for each shuffle, so it differs slightly from the table.