How to Run Pilot Studies: Testing Your Experiment Before the Real Thing
Jumping straight into a full experiment without piloting is like performing surgery without checking the instruments first. Pilot studies catch ambiguous instructions that confuse participants, equipment malfunctions that produce unreliable measurements, ceiling or floor effects in outcome measures, and logistical bottlenecks in scheduling and data collection. Finding these problems with 15 participants is far less costly than discovering them after collecting data from 200.
Define Your Pilot Objectives
Not all pilot studies serve the same purpose, and defining your goals upfront determines the sample size, analysis strategy, and success criteria. Feasibility pilots test whether the study can be conducted as planned. Can you recruit participants at the expected rate? Can the experimental session be completed within the allotted time? Do participants understand the instructions? These questions require qualitative assessment more than statistical analysis.
Parameter estimation pilots collect preliminary data to estimate the expected effect size and variability for the main study power analysis. If no prior data exist for your specific manipulation, a pilot with 20 to 30 participants can provide reasonable estimates. However, effect sizes from small pilots are inherently imprecise and tend to overestimate the true effect, so use them cautiously and consider powering the main study for a smaller effect than the pilot suggests.
Procedural pilots identify logistical problems and refine the experimental protocol. They test timing, equipment, data collection forms, and the flow of the experimental session. Even if you plan to analyze the pilot data, the primary value is experiential knowledge of how the experiment works in practice versus on paper.
Recruit a Small Sample
Pilot studies typically use 10 to 30 participants, depending on the objectives. For feasibility testing, 10 to 15 participants are usually sufficient to reveal major practical problems. For parameter estimation, 20 to 30 provide more stable estimates of means and standard deviations. The sample should match the target population of the main study in relevant characteristics: if the main study will enroll adults aged 40 to 60 with type 2 diabetes, the pilot should enroll adults aged 40 to 60 with type 2 diabetes, not convenience-sampled college students.
Whether pilot participants can be included in the main study is debated. If the pilot led to no changes in the protocol, including pilot participants in the main study increases the total sample size at no cost. If the pilot resulted in procedural changes, pilot participants experienced a different protocol and their data may not be comparable. The safest approach is to treat pilot and main study participants separately unless the protocols are identical.
Run the Full Protocol
Execute every step of the planned experiment exactly as you intend to run it in the main study. Use the same recruitment procedures, the same informed consent forms, the same equipment, the same instructions, the same measurement instruments, and the same data recording methods. The goal is to simulate the main study as faithfully as possible so that any problems that emerge in the pilot would also emerge in the main study.
Take detailed notes during each pilot session. Record how long each phase takes, which instructions caused confusion, where participants hesitated or asked questions, whether the equipment worked reliably, and any unexpected events. These qualitative observations are often more valuable than the quantitative data because they reveal problems that statistical analysis cannot detect.
If possible, debrief participants after the session. Ask them what was confusing, what they thought the experiment was about, whether they experienced fatigue or discomfort, and whether any aspect of the procedure was unpleasant or unclear. Participant feedback provides direct insight into demand characteristics, comprehension, and the overall experience of the experimental protocol.
Analyze and Revise
Analyze the pilot data with the same statistical methods planned for the main study. This tests whether the analysis pipeline works, the data are in the expected format, and the results are interpretable. Do not conduct hypothesis tests on pilot data and interpret them as evidence for or against the treatment effect. The pilot is too small for reliable statistical inference. Instead, use the pilot data to estimate the standard deviation of the dependent variable, check for ceiling or floor effects, and verify that the outcome measure has adequate variability to detect differences between conditions.
Revise the protocol based on what you learned. Simplify confusing instructions, fix equipment problems, adjust the timing of experimental sessions, change outcome measures that showed ceiling or floor effects, and modify any procedures that participants found burdensome or confusing. Document every change and the reason for it. If the revisions are substantial, consider running a second pilot to verify that the changes solved the identified problems.
Use the pilot data to refine your power analysis for the main study. The standard deviation from the pilot provides a more accurate estimate of variability than published values from different populations or procedures. Plug this estimated variability, along with the smallest meaningful effect size, into the power formula to determine the sample size for the main study. Remember that pilot effect sizes are upwardly biased, so power the main study for a more conservative effect.
What a Pilot Study Cannot Do
Pilot studies are frequently misused as underpowered versions of the main study, with researchers testing hypotheses on a small sample and treating the results as preliminary evidence for or against the expected effect. This practice is misleading because pilot samples are too small to provide reliable effect size estimates. A pilot study with 20 participants might produce an effect size of d = 0.8 by chance, leading the researcher to expect a large effect and plan an inadequately small main study. Alternatively, the pilot might produce d = 0.1 by chance, leading the researcher to abandon a study that would have found a meaningful effect with adequate power.
The primary purpose of a pilot study is to test procedures, not hypotheses. A well-designed pilot answers questions like: Can participants understand the instructions? Does the randomization procedure work correctly? Are the measurement instruments producing usable data? How long does each session take? What is the expected dropout rate? These procedural questions are answerable with small samples because they do not depend on statistical power.
When pilot data are used to estimate the expected effect size for power analysis, this estimate should be treated with extreme caution and supplemented with other sources of information, including published effect sizes from similar studies, theoretical predictions, and the smallest effect size that would be practically meaningful. Relying exclusively on a pilot-derived effect size for sample size planning is one of the most common methodological errors in experimental research.
Pilot studies are inexpensive insurance against expensive mistakes. A small investment of time and participants upfront can prevent months of wasted effort by revealing problems that are invisible in the planning stage but obvious in practice.