Power Analysis

What statistical power means

Statistical power is the probability that a test rejects the null hypothesis when a specified alternative is true: the chance of detecting an effect of a given size, if it exists. A study with 80% power for an effect of d = 0.5 would detect such an effect in about 80 of every 100 replications, and miss it in about 20.

Power is a property of a planned study under stated assumptions: the test, the effect size, α and the sample size. It isn't the probability that a study's conclusion is correct.

Type I error

A Type I error is rejecting a null hypothesis that is true: concluding there is an effect when there isn't one. The significance level α is the rate of Type I errors you accept when the null hypothesis is true.

Type II error

A Type II error is failing to reject a null hypothesis that is false: missing an effect that exists. Its probability, β, depends on how large the effect is. A small study can have a high β for realistic effects, so a non-significant result from it says little.

Power = 1 − β

Power is the complement of β: if a design has a 20% chance of missing an effect of the specified size (β = 0.20), its power for that effect is 1 − 0.20 = 0.80. Cohen (1988) suggested 0.80 as a conventional default when there is no other basis for choosing; many funders and journals now expect 0.80 or 0.90, chosen and justified for the study.

Alpha

α, the significance level, is the Type I error rate you choose before analysing data, usually 0.05. It isn't the probability that the null hypothesis is true, and it isn't the probability that a significant result is wrong. A stricter α, such as 0.01, makes Type I errors rarer but needs a larger sample for the same power.

Effect size

The effect size states how large an effect you want to be able to detect, in a form the calculation can use. Each design has its own measure:

Cohen (1988) offered conventional benchmarks, such as d = 0.20, 0.50 and 0.80 for small, medium and large effects. He intended them as a last resort when nothing better is known. A meaningful effect depends on the field, the measures and the practical consequences; the best basis is previous research, a pilot study or the smallest effect that would matter.

Sample size

For a given test, α and effect size, power rises with sample size. Power analysis turns that relationship around: given the power you want, it finds the smallest sample that reaches it. ResearchKit always rounds that sample size up to a whole number, never down, so the target power is met.

A priori power analysis

An a priori power analysis is done before data collection, to decide how many participants a study needs. You specify the test, the effect size to detect, α and the target power, and the analysis gives the required sample size. This is the main use of power analysis, and the one research protocols, ethics applications and funding proposals usually ask for (Faul et al., 2007).

Power for common tests

Power depends on the test, because each test's statistic has its own distribution when the effect exists. ResearchKit calculates t tests and one-way ANOVA exactly from the noncentral t and F distributions, as G*Power does (Faul et al., 2007), and correlations and proportions with standard normal approximations.

One-sample means

A one-sample t test compares a group's mean with a fixed reference value, such as a published norm. The effect size is d = (μ − μ₀) ÷ σ. Power comes from the noncentral t distribution with N − 1 degrees of freedom and noncentrality δ = d√N.

G*Power's manual gives a worked example: a one-sided test of d = 0.625 at α = 0.05 with 95% power needs N = 30. ResearchKit gives the same N and the same actual power, 0.955.

Two independent means

An independent-samples t test compares the means of two separate groups. With d = (μ₁ − μ₂) ÷ σ and equal groups of n, power comes from the noncentral t distribution with 2n − 2 degrees of freedom and δ = d√(n ÷ 2). ResearchKit assumes equal group sizes; unequal allocation needs more participants in total for the same power and isn't supported.

Paired means

A paired design measures the same participants twice, or matched pairs, and is analysed through the differences within each pair. Its effect size, dz, is the mean difference divided by the standard deviation of the differences. Because pairing removes stable differences between participants, dz is often larger than d for the same mean difference, and a paired design can need far fewer participants.

A paired design must not be planned as two independent groups. dz depends on the correlation between the two measurements, so it can't be read off Cohen's benchmarks for d.

Proportions

For a single proportion compared with a fixed value, or two independent proportions, ResearchKit uses Cohen's arcsine method (Cohen, 1988), as the R pwr package does. The effect size h = 2 arcsin √p₁ − 2 arcsin √p₂ treats differences near 0 or 1 as larger than the same difference near 0.5: 0.10 against 0.05 is easier to detect than 0.55 against 0.50.

The method is a normal approximation. With small expected counts, fewer than about 10 successes or failures, it can be inaccurate, and the tool says so; exact methods, such as those in G*Power, are then preferable. Enter proportions as decimals, such as 0.65, not percentages.

Correlation

For a Pearson correlation, the effect size is the correlation r you expect, tested against a null value, usually 0. ResearchKit uses Fisher's z approximation, the large-sample method described in G*Power's manual: the test statistic is normal with mean (z(r) − z(r₀))√(N − 3), where z(r) = ½ ln((1 + r) ÷ (1 − r)).

A correlation is not a standardized mean difference: r = 0.30 is not the same effect as d = 0.30. For very small samples the approximation can differ noticeably from exact methods.

One-way ANOVA

A one-way between-groups ANOVA compares three or more group means. Its effect size, Cohen's f, is the standard deviation of the group means divided by the standard deviation within groups. With k equal groups and N participants in total, power comes from the noncentral F distribution with k − 1 and N − k degrees of freedom and λ = f²N.

G*Power's manual gives a worked example: 10 groups and f = 0.25, at α = 0.05 with 95% power, need 390 participants, 39 per group. ResearchKit gives the same result. The calculation covers the overall F test only, not follow-up comparisons, factorial designs or repeated measures.

One-sided versus two-sided tests

A two-sided test asks whether the parameter differs in either direction. A one-sided test specifies the direction in advance and can only detect an effect in that direction, so it needs fewer participants: for d = 0.5, 102 instead of 128 for 80% power.

The direction must be decided before seeing the data and justified by the research question. Switching to a one-sided test after looking at the results doubles the effective α and misrepresents the evidence.

Sample allocation

For a fixed total, equal group sizes give the most power. ResearchKit's two-group and ANOVA calculations assume equal groups and report both the total and the size of each group. If your groups will be unequal, for example because one is harder to recruit, plan with software that models the allocation ratio.

Minimum detectable effect

When the sample size is fixed, for example by the available population or budget, a power analysis can work the other way: what is the smallest effect this study could detect with the power you want? With 50 participants per group, a two-sided independent-samples t test at α = 0.05 has 80% power for effects of about d = 0.566 or larger. If plausible effects are smaller than that, the study is likely to miss them.

Why effect size matters

The required sample size depends more on the effect size than on anything else, and it is the assumption most open to optimism. Overestimating the effect leads to an underpowered study. Base it on the smallest effect that would matter, or on cautious estimates from earlier work, rather than on the largest effect reported.

Why larger samples increase power

A larger sample estimates means, proportions and correlations more precisely, so the test statistic's distribution under the alternative moves further from zero relative to its spread. The noncentrality grows with the square root of the sample size: δ = d√N for one sample. More participants make the same effect easier to tell apart from no effect.

Why smaller effects require larger samples

Because the noncentrality grows with the square root of N, halving the effect size roughly quadruples the sample needed. For two independent means at 80% power and α = 0.05, d = 0.8 needs 52 participants, d = 0.5 needs 128 and d = 0.2 needs 788.

Assumptions

• The test you plan to use, analysed as planned.
• The effect size, which is an assumption about the population, not a known value.
• The distributional assumptions of the test: for t tests and ANOVA, roughly normal outcomes with equal variances.
• Independent observations, and pairs that really are pairs.
• Equal group sizes, for two-group designs and ANOVA.
• No loss of participants: increase the sample for expected dropout and missing data.

Practical significance versus statistical power

A well-powered study can detect effects too small to matter; an effect can matter even when a study lacks the power to detect it. Power is about detecting an effect reliably. Whether the effect is important is a separate judgement about its size and consequences, and it should guide the effect size you plan for.

Post hoc and observed power

Observed, or post hoc, power is power calculated after a study from the effect it observed. It adds no information: it is a direct function of the p-value, so a non-significant result always has low observed power (Hoenig & Heisey, 2001). It can't show whether a non-significant result reflects no effect or too small a study.

After a study, report the estimated effect size with a confidence interval: the interval shows which effects the data are compatible with. ResearchKit's power-for-a-sample mode calculates power for an effect you specify, such as the smallest effect that would matter; it is labelled as planned power, not observed power.

Reporting power analysis in research

Report the power analysis in your methods section, with enough detail for a reader to repeat it:

• the test the analysis was for, and whether it was one- or two-sided;
• the effect size assumed, and where it came from;
• α and the target power;
• the resulting sample size, per group where relevant, and any allowance for attrition;
• the software or tool used.

A template: “An a priori power analysis for a two-sided independent-samples t test indicated that a minimum of [N] participants ([n] per group) was required to detect an effect of d = [effect size] with [power]% power at α = .05.” Fill in your own values, and give the source of the effect size.

Common mistakes

How to use ResearchKit Power Analysis

1. Choose the analysis you plan to run.
2. Choose what to calculate: required sample size (a priori), power for a planned sample, or the smallest detectable effect.
3. Enter the effect size, or proportions or a correlation, with α, the target power and the test direction.
4. Press Calculate. Read the result, what it means, and the assumptions and method.
5. Use the reporting sentence as a starting point for your methods section.

• Open Power Analysis
• Choose a statistical test
• Calculate an effect size
• Plan a survey sample size

References

The t test and ANOVA calculations follow G*Power (Faul et al., 2007; Faul et al., 2009), and the effect size conventions and arcsine method follow Cohen (Cohen, 1988).

• Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
• Faul, F., Erdfelder, E., Buchner, A., & Lang, A.-G. (2009). Statistical power analyses using G*Power 3.1: Tests for correlation and regression analyses. Behavior Research Methods, 41(4), 1149–1160. https://doi.org/10.3758/BRM.41.4.1149 (external link)
• Faul, F., Erdfelder, E., Lang, A.-G., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences. Behavior Research Methods, 39(2), 175–191. https://doi.org/10.3758/BF03193146 (external link)
• Hoenig, J. M., & Heisey, D. M. (2001). The abuse of power: The pervasive fallacy of power calculations for data analysis. The American Statistician, 55(1), 19–24. https://doi.org/10.1198/000313001300339897 (external link)

Frequently asked questions

Related guides

• How to choose a statistical test

  A practical, step-by-step guide to matching a research question, variables and study design with statistical tests, assumptions and interpretation.

  Read guide

• Confidence Intervals

  What a confidence interval is and isn't, how standard errors, critical values and margins of error produce one, how intervals for means, proportions, their differences and correlations are calculated, and how to interpret and report them.

  Read guide

• What a p-value tells you

  What a p-value measures, what it doesn't, the most common misinterpretations, why statistical significance isn't importance, and how to report and read p-values alongside effect sizes and confidence intervals.

  Read guide

Related tools

• Sample Size Calculator

  Works out a sample size from your assumptions, shows every step of the working, and explains what each assumption means.

  Open tool

• Statistical Test Finder

  Helps you identify statistical tests that may suit your research question and data, and explains why each may fit.

  Open tool

• Effect Size Calculator

  Calculate selected standardized effects from reported summary statistics, with formulas and assumptions shown.

  Open tool

• Power Analysis

  Plans the sample size a study needs, or the power it will have, for seven common designs, with every assumption stated.

  Open tool