1What the method is for

It settles how a result that did not reach significance should be read.

When a test does not reach significance, one must not write that there is no effect. A test answers only whether the null hypothesis can be rejected, and failing to reject it does not mean the null is true. With a small sample, a real effect may still fail to be rejected.

Power is the quantity that draws this distinction. It is defined as the probability of detecting an effect when one is genuinely there. If a high-powered design finds nothing significant, one may say the effect is small. If the design was low-powered, nothing has been said at all. Which of the two it is needs to be shown alongside the interpretation of the result, not before the result is seen.

In this research the method was used to separate absence of effect from smallness of sample, in response to a reviewer's comment that an effect was "not credible".

2Definition

The complement of the type II error. Determined by three quantities.

Definition of power
\[ 1-\beta \;=\; \Pr\bigl(\text{reject the null} \;\bigm|\; H_1 \text{ is true}\bigr) \]

\(\alpha\) … type I error. The probability of saying there is an effect when there is none
\(\beta\) … type II error. The probability of saying there is none when there is one (a miss)
\(1-\beta\) … power. The probability of not missing it

Power is determined by the sample size \(n\), the true effect size, and the significance level \(\alpha\). For a simple design it can be written analytically. For a one-sample \(z\) test, for instance, it takes the following form.

When it can be written analytically (one-sided \(z\) test)
\[ 1-\beta \;=\; \Phi\!\left(\frac{|\delta|\sqrt{n}}{\sigma} - z_{1-\alpha}\right) \]

\(\delta\) … the size of the true effect
\(\sigma\) … the standard deviation
\(\Phi\) … the cumulative distribution function of the standard normal
How to read it … as \(n\) grows the term inside the bracket grows and power rises. A small effect has to be compensated for with \(n\)

Fix any three of the four quantities and the fourth follows

Power, sample size, effect size and significance level are bound to one another. In planning a study the convention is to set power at 0.8 and solve backwards for the required \(n\); but in observational research, where the sample size is settled in advance, what is left is to evaluate afterwards what that \(n\) is capable of detecting. This study is of the latter kind.

3Origin

Neyman and Pearson built the framework; Cohen carried it into practice.

In 1933 Jerzy Neyman and Egon Pearson formulated hypothesis testing as a decision problem balancing two kinds of error (Neyman & Pearson 1933). The most efficient test is the one that minimises the type II error while holding the type I error at a fixed level — and it is here that power was explicitly defined.

In practice, however, attention went to the type I error alone for a long time. In 1969 the psychologist Jacob Cohen wrote Statistical Power Analysis for the Behavioral Sciences, bringing power analysis to researchers in the behavioural sciences. Cohen's \(d\), the widely used index of effect size, belongs to the same current.

Computing changed the method

Analytical power calculations are available only where both the null and the alternative distribution of the test statistic can be written down. For hierarchical and mixed models, often they cannot be written in closed form. As computing became cheap, generating data and counting became a general method. Simulation-based power calculation is the product of that.

4Cohen's warning, and the replication problem

Mass-produce low-powered studies and the literature as a whole stops being trustworthy.

In 1962 Cohen examined the 78 papers published in the 1960–61 volumes of the Journal of Abnormal and Social Psychology and computed power for the 70 of them that contained tests bearing on a hypothesis. The result was that the probability of obtaining statistical confirmation for a medium effect was on average less than one in two (Cohen 1962). Sample sizes were being fixed by convention, convenience and negotiation, not by any reasoned power analysis.

Low power is not only a problem of missed effects

In a low-powered design, the results that do reach significance are distorted as well. When an effect is small, only the samples that happen to come out large cross the threshold, so the effect sizes that get reported are larger than the true value. If non-significant results are then harder to publish, what remains in the literature is nothing but inflated estimates. Publication bias combined with low power shifts a field's estimates systematically, without any misconduct in the individual studies.

This structure is why power comes up repeatedly in discussions of the replication problem. A failed replication does not necessarily mean the original study was false. Often it is enough to say that the original design was not at a level that could capture the effect reliably.

5Design-based simulation

Rather than substituting into a formula, generate data from the actual design and count.

The model in this study is a beta-binomial mixed model with random effects at three levels: item, settlement and district. No analytical power formula exists. Data are therefore generated using the actual design as it stands.

  1. Fix the design　Use the real number of middle-school students and older-generation respondents in each settlement, the real standardised population growth rate and size, 102 items and 25 settlements, exactly as they are. The point is to carry the real imbalance across rather than construct a hypothetical balanced design.
  2. Borrow the variance structure　Take the variance components of the random effects from fit_rawcount, the count-level model fitted to the real data.
  3. Set a true effect　Fix the true value of the generation-by-growth interaction at \(\beta_{\text{true}} \in \{0,\,-0.15,\,-0.30,\,-0.50\}\).
  4. Generate　Generate beta-binomial data under that true value. One hundred datasets at each level.
  5. Re-estimate　Estimate the generated data with a fast generalized linear mixed model (lme4, nAGQ=0, overdispersion approximated by an observation-level random effect).
  6. Count　Take the proportion of runs in which the 95% confidence interval on the interaction fell below zero as the power. The distribution of the estimates is recorded alongside.
Counting power as a proportion
\[ \widehat{1-\beta} \;=\; \frac{1}{S}\sum_{s=1}^{S} \mathbf{1}\bigl\{\,\text{CI}^{(s)}_{\text{upper}} < 0 \,\bigr\}, \qquad S=100 \]

\(\mathbf{1}\{\cdot\}\) … 1 if the condition holds, 0 otherwise
How to read it … the proportion of the hundred generated datasets in which the interaction could be said to be genuinely negative. This is the estimate of the power of this design

This estimate has error of its own

With a hundred runs per level, the power estimate is itself a sample value following a binomial distribution. If the true power is around 0.12, the standard error over a hundred runs is roughly 0.03. The figures in the table should not be read as settled to two decimal places.

6Results in this research

The estimator is sound. What was short was the power.

\(\beta_{\text{true}}\)Mean of estimates90% rangePower
0−0.015[−0.284, 0.254]0.02
−0.15−0.147[−0.447, 0.179]0.12
−0.30−0.317[−0.636, −0.066]0.39
−0.50−0.541[−0.876, −0.257]0.82

Figures quoted directly from power_simulation.csv, the output of 20_power_simulation.R. One hundred simulations at each level.

The estimator is unbiased

The mean of the estimates recovers the true value almost exactly (−0.147 against −0.15, −0.317 against −0.30, −0.541 against −0.50). The model is not returning biased estimates. The problem lies in the size of the spread: the 90% range is wide at every level, and at −0.15 and −0.30 it straddles zero.

At realistic effect sizes, almost nothing can be detected

Over the range of effect sizes anticipated in research on language change — from −0.15 to −0.30 — power stays between 12% and 39%. Repeat this study a hundred times and the effect would be captured in between 12 and 39 of them. Power reached 0.8 only when an effect as large as −0.50 was set.

Placing the observed value on this distribution

The estimate of the interaction in the real data is −0.165. That is consistent with the distribution obtained when \(\beta_{\text{true}}\) is set somewhere between −0.15 and −0.20, and falls outside the centre of the distribution under no effect (−0.015). The observation is therefore more naturally read as a medium-sized effect for which power was insufficient.

What this result changed is the wording of the claim

A power simulation does not prove that an effect exists. All it can do is rule out the reading that there was no effect. In this research the main result was rewritten as "the direction is consistent and suggestive, but not decisive given the limits of the design's power", stated alongside both the agreement in direction between the aggregate and individual-level models and the low power of the design.

It also became clear that the high precision shown by the aggregate model (posterior probability 0.99) was the result of ignoring the uncertainty arising from the very small number of older-generation respondents (a median of a few people). The power evaluation has knock-on effects on how the other results are read.

7Sources and data

The simulation is implemented in 20_power_simulation.R, which runs reconstruction of the real design, generation, re-estimation and tabulation in a single pass. The inputs are the analysis master and the settlement list; the outputs are power_simulation.csv and the figure FigS_power. The analysis is laid out as R scripts, and the original data files are never modified — they are read only through working copies.

References for the method

  • Neyman, J. & Pearson, E. S. (1933) On the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London, Series A, 231, 289–337. doi:10.1098/rsta.1933.0009
  • Cohen, J. (1962) The statistical power of abnormal-social psychological research: a review. Journal of Abnormal and Social Psychology, 65(3), 145–153. doi:10.1037/h0045186
  • Cohen, J. (1969) Statistical Power Analysis for the Behavioral Sciences.

Which perspective the method belongs to

Power measures how often a procedure comes out right when the sample is drawn again, with the true value held fixed. Like the type I error, it is defined as a proportion across repetitions. What this simulation does is count that proportion directly by generation, for a design whose analytical formula cannot be written. The picture set out under perspective 2 (frequentist) in Read the seven perspectives in equations — that the probability attaches to the interval while the parameter does not move — applies here as it stands.

This page sets out a method Onohara uses in actual research, written for readers in the same field. All figures are quoted from the output of analysis scripts; no estimates or approximations are given.
For the whole picture of where the methods sit, see Read the seven perspectives in equations; for which course sessions fall under which perspective, see Methods & Mathematics.