We can regularly encounter a client trying to answer a question that is so easy to understand and at the same time so significant: What is the number of participants that I will require in my study?
It is essential to get the sample size correct. Smaller, and you may find inconclusive or misleading results. Excessively large and you squander resources, time and money. This guide skips all the technicalities of the statistics to give you an idea of the basics and make an informed choice.
The Importance of Sample Size: Statistical Significance
When conducting research, you typically do not research the entire population (e.g., all customers, all patients, all devices). Rather, a sample- a smaller and representative group is studied. You want to ensure that your sample findings can support secure inferences about the rest of the population. You have trained on data up to October 2023.
Conceptualization of Statistical Significance
The statistical significance concept assists us in knowing whether the effect we have detected or the difference we have detected in our sample is likely to be a real effect in the population or it is merely by chance.
- The Null Hypothesis: A null hypothesis is the starting point of every study and generally the null hypothesis is a statement that there is no effect or no difference (e.g., The new ad campaign has no effect on sales.)
- The p-value: Statistical tests will compute the p-value after gathering data. This value will be the chances of seeing your results (or something more drastic) in case the null hypothesis is true.
- The Significant Threshold: Researchers generally employ a value of 0.05 (or 5 per cent). When your p-value is below that 0.05 (p<0.05), then you reject the null hypothesis. You find the result to be statistically significant and probable to be true.
This is because it is hard to get a small p-value even with a true effect due to the small sample size. A false negative- false positive error results, which we refer to as a Type II Error.
The Effect Size and Role of Power
Researchers require a study of adequate statistical power in order to prevent this expensive error.
- Statistical Power is the likelihood of rejecting the null hypothesis when the true effect is real. The power of 80% (0.80) is generally desired by researchers.
- Effect Size is the size of your difference or relationship you are seeking. When your effect is anticipated to be large (e.g. a new drug that will cure 80% of people), you will require a smaller sample. It will take a significantly bigger sample to be able to reliably identify a small, subtle effect (e.g. a small increase in website clicks).
In essence, a Power Analysis estimates the required sample size by balancing your required Power, Significance Level (usually 0.05), and your anticipated Effect Size. This will ensure that your study is well prepared to identify what you aimed to detect.
The Reality Check: The Hard Facts
Researchers may indicate the use of a huge sample size in the theory of statistics, but in reality, they are always limited by practical considerations. You have to reconcile statistical rigour and viability.
Budget and Time
All the participants recruited, tested, or surveyed cost money and time.
Action Step: Be sensible when it comes to financial resources and project time. A small, well-done study that researchers complete in good time is usually superior to a grandiose statistically ideal study that no one ever finances or writes.
Accessibility and Population of Interest
How simple is the process of locating your participants?
Hard-to-Reach Groups: When you are researching a rare disease or a group of high-profile professionals, or a highly specific demographic, you have a small possible participant pool. You must use the pool that you have, and you may have a smaller sample, so you should consider this in your report. You are trained on data up to October 2023.
Broad Consumer Groups: When you are looking at the general consumers, recruitment is not a problem, and you need to target a bigger and stronger sample.
Data Quality and Study Design
The nature of research you are doing will also influence the size of the sample that you need.
- Qualitative Research (Interviews, Focus Groups): These are studies that are more in depth than breadth. When you no longer get new information on the interviews you have you usually stop at the saturation point of data. This can typically have a much smaller sample (10-30 participants).
- A/B Testing or Complex Experiments: They involve balanced samples in a careful way and they tend to have larger samples to consider the various variables and noise in the data.
Conclusion: A Strategic Approach to Sample Size
The appropriate sample selection is not merely a mathematical choice. The sophistication should not paralyze your research.
The following is your simple action plan:
- Determine the Target Effect: Figure out the minimal effect size that would be of practical interest to your business or research problem. (e.g. we only care whether the new feature will increase conversions by at least 5%).
- It is suggested to consult a Statistician: You should use a Power Analysis with the desired effect size, the desired power (80%), and a desired level of significance (α=0.05) to calculate the statistically required sample size.
- Use the Reality Filter: Reconsider the number based on your budget, time, and access to respondents.
When the statistically required size is too large, you can choose between increasing resources or accepting the fact that you possibly can only find larger effects with your current resources.
Are you now ready to take your research off the ground? Fast Publication will guide you with the hassles of calculating the sample size, whereby your study is not only statistically correct but also practical.
Is there a particular research question that you are not sure about? Share it with us in the comments, or come and see us today and get a personalized consultation!
