1What the method is for
A comparison of richness does not hold unless sample sizes are levelled.
Write \(S_s\) for the number of kinds observed in group \(s\). \(S_s\) depends far less on "how many kinds are there" than on how much was surveyed. Survey more individuals and the chance of meeting a kind not met before rises, so \(S_s\) increases monotonically with the sample size \(N_s\). An observation that \(S_A > S_B\) is therefore no evidence of a difference in richness so long as \(N_A > N_B\).
This relation is a probabilistic necessity, not a defect of measurement. The rarer kind \(i\) is in the population, the more easily it is missed in a small sample. Increase the sample size and fewer are missed. \(S_s\) rises by exactly that much. The observed number of kinds appears as a product of richness and survey effort.
Rarefaction removes this confounding by calculation. For every group it asks what number of kinds would be expected to appear had only \(m\) individuals been drawn at random, and compares under a common \(m\). Nothing is actually discarded from the sample; the expectation is computed analytically.
2Origin — who made it, and in which field
From research that set out to compare biodiversity on the deep-sea floor and found it could not be compared.
Howard Sanders proposed it in The American Naturalist in 1968. Working on deep-sea benthic communities at the Woods Hole Oceanographic Institution, he was trying to compare the diversity of samples taken from environments differing in latitude, depth, temperature and salinity, and ran into the problem of the previous section. If the number of individuals collected differs by location, species counts cannot be compared as they stand. Sanders brought in the idea of thinning samples down to a common size, and called it rarefaction.
Sanders's first formulation carried a computational bias and was found to overestimate species numbers. Stuart Hurlbert corrected it (1971, Ecology). The title of that paper is "The nonconcept of species diversity", and its subject was a critique: the debate around diversity indices was confused both semantically and technically. As an alternative Hurlbert offered several quantities with settled biological interpretations, one of which is the expected number of species. The formula used as rarefaction today is Hurlbert's.
The method was born as a critique of diversity indices
This provenance bears on how it should be read. What Hurlbert offered was not an index for collapsing diversity into a single number but a plainer quantity with a settled interpretation: count the species after levelling the sample size. To use rarefaction is also to take the position of avoiding the abstraction "diversity" and building comparable conditions before counting anything.
3Where it comes from mathematically
The hypergeometric distribution. Seventeenth- and eighteenth-century combinatorics, used as it stands.
Suppose that of \(N\) individuals, \(N_i\) belong to kind \(i\). Draw \(m\) individuals at random without replacement: when does the drawn sample contain not one individual of kind \(i\)? Exactly when all \(m\) were chosen from the \(N-N_i\) individuals that are not of kind \(i\).
The denominator is the total number of ways to choose \(m\) from \(N\); the numerator counts those that avoid kind \(i\). This is the probability of drawing zero under a hypergeometric distribution — the same structure as the classical question of how many red balls come out when \(m\) are drawn from a bag of red and white.
Take the complement and it becomes the probability that kind \(i\) appears at least once in the sample. What is wanted is the expected total number of kinds observed, so write it as a sum of indicator variables and add them up by linearity of expectation.
\(S\) … total number of kinds observed across the whole sample
\(N\) … the sample size of that group, \(N_i\) … number of individuals of kind \(i\), \(m\) … the depth to level at
Linearity of expectation … the per-kind appearance probabilities are simply added. Independence between kinds is not assumed
It is a closed formula, not a simulation
Repeatedly thinning the sample at random and counting would converge to the same value, but there is no need. The formula above gives the expectation itself. In implementation it is stable to compute \(\binom{N-N_i}{m}\big/\binom{N}{m}\) in logarithms and exponentiate (exp(lchoose(N-k, m) - lchoose(N, m))); when \(N-N_i < m\) this automatically becomes 0 and that kind contributes 1.
4Transfer to language research — what is being read across
An operation replacing the ecologist's "individual" and "species" with the survey's "speaker" and "vocabulary item".
Research on cultural evolution often takes the relation between population size and cultural diversity as its theme. For vocabulary the question becomes whether larger settlements have richer vocabulary. The number of vocabulary items observed here carries a problem of exactly the same structure as species counts in ecology. The larger the settlement, the more speakers can be surveyed, so the more items are observed.
Number of individuals collected \(N\) → number of speakers surveyed
Number of individuals of kind \(i\), \(N_i\) → number of speakers \(k_i\) answering that they use vocabulary item \(i\)
Observed species count \(S_m\) → number of vocabulary items expected to appear when \(m\) speakers are surveyed
The form of the equation does not change. Only \(N_i\) is read as \(k_i\).
One point where the structure differs
In the ecological setting, one individual belongs to only one species, so \(\sum_i N_i = N\) holds. In a language survey, however, one speaker uses many vocabulary items. In this study, with 106 items, a single person answers "use" for dozens of them, so \(\sum_i k_i \gg N\).
The formula itself does not break down. All the derivation above uses is the probability that no user of item \(i\) is among the \(m\) speakers, and that does not depend on whether a speaker uses other items as well. It corresponds to an established usage in ecology too, where presence and absence are counted per sampling unit rather than per individual (incidence-based / sample-based rarefaction).
But the interpretation changes. The quantity obtained is not "the expected number of species when \(m\) individuals are collected" but "the expected number of items that at least one person answers they use when \(m\) people are asked". Carry the intuition of individual-based rarefaction across unaltered and this difference goes unnoticed.
5How it was used in this research
Checking whether the correlation between population size and vocabulary richness is an artefact of survey effort.
- Reconstructing individual-level data Rarefaction needs responses per person rather than a summary table, so individual records for middle-school students were reconstructed from the master sheet of the original data. Extracting on the conditions that the school name be real, the year be 1 to 3, and every vocabulary response be 0, 1 or 2 gave 510 people across 49 settlements, with a value of 2 ("use") counted as usage.
- Building the usage matrix For each of the 106 vocabulary items, count the number surveyed \(N\) per settlement and the number of users \(k_i\) of item \(i\). The same tabulation is built by lineage as well (Kansai/Ōmi, Tokyo/standard, Mino/Owari).
- Raw richness Count the items for which \(k_i > 0\). This is richness before correction.
- Rarefied richness Compute \(\mathbb{E}[S_m]\) by Hurlbert's formula. Three depths are tried, \(m = 3,\,5,\,8\). Settlements with \(N < m\) cannot be computed and are excluded (37 settlements remain at \(m=3\), 29 at \(m=5\)).
- Regression on population Regress raw richness and rarefied richness each on the common logarithm of settlement population, and compare the slopes.
| Lineage | Raw richness | Rarefied \(m=3\) | Rarefied \(m=5\) |
|---|---|---|---|
| All items | +21.0 to +21.5 (p<.001) | −4.3 (p=.21) | −2.4 (p=.49) |
| Kansai / Ōmi | +10.3 to +11.5 (p<.001) | −2.8 (p=.23) | −3.3 (p=.22) |
| Mino / Owari | +8.5 to +13.4 (p<.01) | −1.3 (p=.55) | +0.6 (p=.81) |
| Tokyo / standard | +0.7 to +0.8 (p<.03) | −0.3 (p=.38) | +0.1 (p=.68) |
Slopes of the regression on the common logarithm of settlement population. Figures are quoted from run results of the analysis scripts as recorded in the working log. The ranges on the raw side arise because the values shift with the version of the lineage classification.
Once corrected, the association disappears and turns slightly negative
With raw richness the slope against log population is around \(+21\) for the total item count, all with \(p<.001\) — a strong positive association. Level the sample sizes, however, and nothing is significant for any lineage and the sign is not stable. The conclusion agrees at \(m=3\) and \(m=5\). The observation that "settlements with larger populations have richer vocabulary" is almost entirely explained by the difference in survey effort: more middle-school students were surveyed in larger settlements.
The same conclusion comes from a second, independent route. Rather than physically levelling samples as rarefaction does, controlling for the log of the number surveyed as a covariate in a hierarchical model drops the population effect for the Kansai lineage to \(-0.07\). That two procedurally different methods converge on a conclusion in the same direction is the basis for this judgement.
6Limits — what the method cannot say
Levelling survey effort does not remove the ceiling of the measurement.
With a fixed-list survey, the ceiling binds
This is the heaviest constraint. In the ecological setting there is no artificial upper bound on the number of species that can be observed; collect more and there is always room for a new species to appear. This survey, however, was conducted with a checklist of 106 items prepared in advance. The upper bound on richness is 106 and in principle it cannot go higher.
The genuinely richer a settlement is in vocabulary variation, the closer it moves to filling the list. Real differences are then compressed against the ceiling of the list. It is the same structure as measuring differences in ability with a hundred-question test: once the top performers are all near full marks, the differences stop being visible. Rarefaction removes the confounding of survey effort; it does nothing about this ceiling.
The conclusion of this research is therefore a negative one: under a fixed-list survey, richness cannot be used as an index of vocabulary diversity. It did not show that there is no association between population and diversity; it showed that with this method of measurement the presence or absence of an association cannot be judged. Those are different claims.
Assumptions carried in by the transfer
As stated in section 4, a speaker, unlike an individual organism, uses many items at once. Ecological rarefaction moreover presupposes that individuals are drawn at random from the population, whereas this survey was conducted by area, so non-independence within households cannot be ruled out. Middle-school students in the same household are likely to share vocabulary. The effective sample size may be smaller than \(N\), in which case the uncertainty on the rarefied values is larger than it appears.
A reservation about the reconstructed data itself
Because rarefaction requires individual-level data, individual records were reconstructed from already-tabulated source material. Richness obtained from this reconstruction does not match the published values exactly; the correlation is 0.64. The cause of the discrepancy has been confirmed to lie in differences in item classification and in the threshold for judging usage, not in errors in the data. The comparison of raw against rarefied richness is a comparison within a single reconstructed dataset, so the conclusion is internally valid; identifying the original set of individuals themselves would raise the confidence further.
7Sources and data
A secondary analysis of a three-generation dialect vocabulary survey covering five middle-school districts from Maibara in Shiga Prefecture to Sekigahara in Gifu Prefecture. The questionnaire elicited 106 dialect vocabulary variants across 40 question concepts, classified in advance into 45 Kansai/Ōmi forms, 31 Tokyo/standard forms and 26 Mino/Owari forms, with four items of youth speech and new dialect. The original survey was conducted in 2015 by Habiro-Net, and this research analyses the anonymised data. Every stage of the analysis is scripted, and the original data files are never modified — they are read only through working copies.
References for the method
- Sanders, H. L. (1968) Marine benthic diversity: a comparative study. The American Naturalist, 102(925), 243–282. doi:10.1086/282541
- Hurlbert, S. H. (1971) The nonconcept of species diversity: a critique and alternative parameters. Ecology, 52(4), 577–586. doi:10.2307/1934145
- Henrich, J. (2004) Demography and cultural evolution: how adaptive cultural processes can produce maladaptive losses—the Tasmanian case. American Antiquity, 69(2), 197–214. doi:10.2307/4128416 (a representative instance of the argument linking population size to cultural diversity)
Which perspective the method belongs to
What rarefaction addresses is the problem of how a statistic changes with the way the sample is taken. The quantity in the population is treated as fixed, and comparability between observed quantities is restored by levelling the sample size. In computing the expectation directly from combinatorics without assuming a distributional form, it overlaps with perspective 2 (frequentist) in Read the seven perspectives in equations. As with the Mantel test (in Japanese), which builds a null distribution by permutation, the idea is to operate on the sample side to construct the standard for a judgement.
This page sets out a method Onohara uses in actual research, written for readers in the same field. All figures are quoted from the output of analysis scripts and from working records; no estimates or approximations are given.
For the whole picture of where the methods sit, see Read the seven perspectives in equations; for which course sessions fall under which perspective, see Methods & Mathematics.