Module 5: Non-Probability Sampling

Overview

In this module, we will discuss how researchers use non-probability sampling in political science research, and how to critically assess research that uses non-probability samples. We will then discuss research that uses a very small number of cases (“small n research”).

As always, to help you better understand and engage with the ideas that we are covering in the modules ahead, I strongly encourage you to discuss the course material in the class discussion boards.

Objectives

When you have finished this module, you should be able to do the following:

  1. Explain the role of nonprobability sampling in political science research;
  2. Distinguish between five common nonprobability sampling techniques;
  3. Explain why nonprobability sampling is used in some political science online survey research, and identify the questions raised by this sampling approach;
  4. Critically assess the use of non-probability sampling approaches when reading social science research studies;
  5. Explain how case study approaches are used to study politics; and
  6. Explain how comparative approaches are used in political science.

Module Instructions

  1. Read Chapter 8 (pages 168-171) in the 3rd edition of our textbook or Chapter 5 (Pages 101-110) in the 4th edition of our textbook. Create self-study flashcards for the chapter.
  2. Watch mini-lectures “Nonprobability Sampling”, “Nonprobability Sampling Techniques”, “Nonprobability Sampling in Online Survey Research,” and “Critiquing Nonprobability Samples.”
  3. Read Chapter 7 in the 3rd edition of our textbook or Chapter 10 in the 4th edition of our textbook. Create self-study flashcards for the chapter.
  4. Watch mini-lecture “Small n research”.
  5. Complete Learning Activity.

Key Terms and Concepts

  • Coverage bias
  • Non-response bias
  • Non-probability sampling
  • Accidental sample
  • Quota sampling
  • Purposive sampling
  • Snowball (network) sampling
  • Data saturation
  • Self-selection
  • Opt-in sampling
  • Transferability
  • Scope conditions
  • Case study
  • Counterintuitive
  • Descriptive case study
  • Theory testing case study
  • Failed most likely case
  • Successful least likely case
  • Process tracing
  • Comparative research
  • Most similar systems design
  • Most different systems design
  • Equivalent measures

Required Readings

  1. Chapter 7 and Chapter 8 (pages 168-171) in Berdahl, Loleen and Keith Archer. Explorations: Conducting Empirical Research in Canadian Political Science (Third Edition). Oxford University Press OR Chapter 5 (pages 101-110) and Chapter 10 of Berdahl, Loleen and Jason Roy. Explorations: Conducting Empirical Research in Canadian Political Science (Fourth Edition). Oxford University Press.


 

Learning Material

Nonprobability Sampling

Nonprobability Sampling Techniques

Use of Nonprobability Sampling in Online Survey Research

Critically Assessing Nonprobability Samples

Small N research

Learning Activity

Media Story Critique

  1. Watch the CNN news story “Did Seaworld stuff ballots for Blackfish poll?” (https://www.cnn.com/2014/01/03/tech/web/sea-world-blackfish-poll/index.html ). In 300-500 words, explain how this story illustrates some of the challenges of a non-representative sample, and then explain how a non-representative sample is more appropriately used. In your response, use at least two terms covered in the module (readings and/or videos), and be sure that all terminology is used correctly. Proofread carefully.
  2. Post your critique in your Learning Activity Discussion Board.
  3. Provide a constructive response to at least one of your fellow group members’ posts. A constructive response is one that (a) uses supportive language to (b) identify for the author an area in which the work can be strengthened. For example, it may identify an issue where the wording is unclear or a point where terminology is used incorrectly, or suggest ideas for examples or ways to strengthen the argument, or let the author know of questions that the work raised for them. A constructive response goes beyond ‘I agree’ or ‘that is interesting’ to assist the author in improving the work. It should provide feedback that is intended to assist the author of the learning activity in improving their work.

Reminder: At the end of Module 8, you are required to select one learning activity for submission from Modules 5-8. You can use the feedback that you receive in the group forum to revise your selected learning activity prior to submission.

Glossary

accidental sample a non-probability sample technique in which researchers gather data from individuals whom they “accidentally” encounter or who are convenient; also known as a sample of convenience or haphazard sample.

case study a method of analysis that involves an in-depth investigation of a single individual, group, or event; in political science, generally used with the intent of identifying general causal principles.

comparative research a research design that seeks to compare phenomena across different political systems or cultures.

coverage bias type of bias that occurs when a research sampling frame excludes particular groups.

counter-intuitive a condition that occurs when a situation, event, or outcome differs from dominant theoretical expectations or common sense.

data saturation: point at which data collection no longer provides the researcher with new information

descriptive case study a study that aims to chronicle or describe some aspect of reality; involves questions of who, what, where, and when.

equivalent measures indicators that measure the same concept across different countries, political systems, or cultures.

failed most-likely case a case that is expected to confirm the operating assumptions of a particular theory but in practice refutes them.

most-different-systems design a comparative research design in which the researcher compares very different systems in an attempt to explain similarities between them.

most-similar-systems design a comparative research design in which the researcher compares very similar systems in an attempt to explain differences between them.

non-response bias the sampling error that occurs when survey respondents and non-respondents differ in an important way.

Opt-in sampling non-probability sampling technique in which cases are recruited or volunteer to participate in research studies, typically for a cash or other reward

process tracing a research method that generates causal pathways between the independent and dependent variables of a case by connecting a series of observations.

purposive sampling a non-probability sampling technique in which researchers use their judgment to select cases that will provide the greatest amount of information; also known as judgmental sampling.

quota sampling a non-probability sampling technique in which the researcher combines purposive or accidental sampling with stratification; the researcher identifies a number of target groups (strata) and then sets a quota number that must be met for each group.

scope conditions the limits to which particular research can make valid claims.

self-selection a process in which individuals select themselves for participation in a sample (e.g. radio call-in programs).

snowball (or network) sampling a non-probability sampling technique in which the researcher begins by identifying a few cases and, from these, gets referrals for other cases and continues to branch out.

successful least-likely case a case that is expected to refute the operating assumptions of a particular theory but in practice confirms them.

theory testing case study research that seeks to modify an existing theory or generate a new theory.

transferability the extent to which researchers can export the lessons drawn from one investigation to develop conclusions about another set of cases.

Note: Unless otherwise stated, glossary source is Berdahl, Loleen and Keith Archer. 2015. Explorations: Conducting Empirical Research in Canadian Political Science (Third Edition). Oxford University Press.

References

AAPOR. 2013. Report of the AAPOR Task Force on Nonprobability Sampling. Available at https://www.aapor.org/AAPOR_Main/media/MainSiteFiles/NPS_TF_Report_Final_7_revised_FNL_6_22_ 13.pdf

Berdahl, Loleen and Keith Archer. 2015. Explorations: Conducting Empirical Research in Canadian Political Science (Third Edition). Oxford University Press.

Lund Research Limited. 2012. Nonprobability Sampling. http://dissertation.laerd.com/non-probability-sampling.php

Mercer, A., Kreuter, F., Keeter, S. and Stuart, E.A. 2017. “Theory and Practice in Nonprobability Surveys: Parallels between Causal Inference and Survey Inference.” Public Opinion Quarterly, Volume 81: 250– 271.

Source: Rubin, Allen. 2008. Practitioner’s Guide to Using Research for Evidence-Based Practice. New Jersey: John Wiley & Sons Inc.