When power analyses based on pilot data are biased: inaccurate effect size estimators and follow-up bias

C. Albers, D. Lakens

Research output: Contribution to journalArticleAcademicpeer-review

53 Citations (Scopus)

Abstract

When designing a study, the planned sample size is often based on power analyses. One way to choose an effect size for power analyses is by relying on pilot data. A-priori power analyses are only accurate when the effect size estimate is accurate. In this paper we highlight two sources of bias when performing a-priori power analyses for between-subject designs based on pilot data. First, we examine how the choice of the effect size index (η2, ω2 and ε2) affects the sample size and power of the main study. Based on our observations, we recommend against the use of η2 in a-priori power analyses. Second, we examine how the maximum sample size researchers are willing to collect in a main study (e.g. due to time or financial constraints) leads to overestimated effect size estimates in the studies that are performed. Determining the required sample size exclusively based on the effect size estimates from pilot data, and following up on pilot studies only when the sample size estimate for the main study is considered feasible, creates what we term follow-up bias. We explain how follow-up bias leads to underpowered main studies. Our simulations show that designing main studies based on effect sizes estimated from small pilot studies does not yield desired levels of power due to accuracy bias and follow-up bias, even when publication bias is not an issue. We urge researchers to consider alternative approaches to determining the sample size of their studies, and discuss several options.

Original languageEnglish
Pages (from-to)187-195
Number of pages9
JournalJournal of Experimental Social Psychology
Volume74
DOIs
Publication statusPublished - 1 Jan 2018

Keywords

  • Effect size
  • Epsilon-squared
  • Eta-squared
  • Follow-up bias
  • Omega-squared
  • Power analysis

Fingerprint Dive into the research topics of 'When power analyses based on pilot data are biased: inaccurate effect size estimators and follow-up bias'. Together they form a unique fingerprint.

Cite this