Skip to content

Bias in Sampling and Surveys

Identify sources of bias in sampling and surveys including voluntary response and convenience sampling.

Written and reviewed by the Study Mondo Education TeamLast updated
🎯⭐ INTERACTIVE LESSON

Try the Interactive Version!

Learn step-by-step with practice exercises built right in.

Start Interactive Lesson →

⚠️ Bias in Sampling and Surveys

Types of Sampling Bias

Undercoverage

  • Some population members systematically excluded
  • Sampling frame does not include all population members
  • Example: telephone survey misses homeless, institutionalized, no-phone individuals
  • Results biased toward accessible subgroup

Nonresponse Bias

  • People who don't respond differ from responders
  • Higher rates in mail surveys (~50%), lower in in-person (~80%)
  • Cannot be completely eliminated, can be reduced via multiple follow-ups
  • Example: survey on financial habits—wealthy more likely to refuse

Voluntary Response Bias

  • Self-selected responders not representative
  • Highly motivated to respond (strongly agree or disagree)
  • Example: online polls, call-in surveys
  • Results typically more extreme than population opinion

Types of Response Bias

Wording Effects

  • Question phrasing influences answer
  • Example: "Should government spend more on welfare?" vs "Should government spend more on helping poor?"
  • Leading questions subtly suggest preferred answer
  • Double negatives confuse respondents

Interviewer Effects

  • Interviewer appearance, tone, demographics influence response
  • Subjects give socially desirable answers (social desirability bias)
  • Example: respondent gives less honest answer to attractive interviewer
  • Reduced by trained, neutral interviewer

Question Order Effects

  • Previous questions prime respondents
  • Example: ask about healthcare costs, then satisfaction → answers affected
  • Randomize or pilot-test question order

Timing and Setting

  • When/where survey taken affects response
  • Example: customer satisfaction survey in store vs. email (both environments bias)

Strategies to Reduce Bias

BiasReduction
UndercoverageUse comprehensive sampling frame; multistage or stratified to reach hard-to-reach groups
NonresponseMultiple contact attempts, incentives, follow-up with non-respondents
Voluntary responseUse random sampling instead
WordingPilot-test questions; use neutral language; avoid leading questions
InterviewerTrain interviewers; use telephone/email; randomize interviewer assignment
Social desirabilityEnsure anonymity; use indirect questions (third-party phrasing)

Common Mistakes

  • Assuming larger sample size eliminates bias (bias is fixed-size problem, not variability)
  • Confusing bias (systematic) with variability (random)
  • Believing no-response survey can be unbiased without follow-up

Decision Rule

Is bias fixed (doesn't decrease with larger nn)? → Reduce via improved design Is variability decreasing with nn? → Increase sample size

AP Exam Tip

When critiquing a survey, identify: potential source of bias (which type), explain mechanism, and propose mitigation strategy. Larger samples fix variability, not bias.

📚 Practice Problems

1Problem 1easy

❓ Question:

A survey asking 'Don't you agree that more funding should go to schools?' is an example of what type of bias? Explain what response bias is and how to fix it.

💡 Show Solution

This is an example of response bias (specifically leading question bias or wording bias).

What response bias means: Response bias occurs when question wording, question order, or surveyor behavior influences responses, causing answers to differ from true opinions.

Why this question is biased: The phrasing 'Don't you agree...' is leading — it suggests the desired answer is 'yes.' Most respondents will comply rather than disagree with a leading question, inflating support for school funding.

How to fix it:

Reword as neutral and balanced alternatives:

  1. 'How should school funding change?' (open-ended)
  2. 'Do you support increasing, maintaining, or decreasing school funding?' (balanced options)
  3. 'School funding should be increased' vs. 'School funding should be maintained' (state both sides)

Key principle: Survey questions must be neutral and not suggest a particular answer. Both sides of an issue should be presented equally.

Other response bias types: Acquiescence bias (always agreeing), social desirability bias (answering what sounds good, not true), and question order effects.

2Problem 2medium

❓ Question:

An online poll asks: 'Have you voted in the last election?' Those who don't respond are removed from the survey. What biases might this introduce?

💡 Show Solution

Biases introduced:

1. Nonresponse bias:

People who don't respond differ from respondents. Who doesn't respond?

  • Busy people (less time to fill out surveys)
  • Less interested in voting/politics
  • Elderly or tech-unfamiliar people (online survey may exclude them)
  • People with contrary opinions (may ignore it)

Result: Sample overrepresents voters and politically engaged people; underrepresents non-voters.

2. Voluntary response bias (self-selection):

Online polls are voluntary. Who volunteers?

  • People with strong opinions (both for and against)
  • More motivated individuals
  • Those with internet access

Result: Response represents passionate people, not average population.

3. Undercoverage (selection bias):

Only people with internet access can take online survey. Missing:

  • Elderly, rural poor (limited internet)
  • Low-income households
  • Tech-averse individuals

These groups have different voting patterns than internet users.

4. If the question assumes voting:

'Have you voted...?' assumes respondents are voters. People who think they shouldn't answer (non-voters, non-citizens) might skip the survey, creating further nonresponse bias.

Consequences:

Estimated voter turnout would be inflated — the poll would overestimate how many people actually voted.

How to reduce bias:

  1. Use random sampling (not voluntary signup)
  2. Use multiple data-collection methods (phone, mail, in-person) to reach diverse groups
  3. Offer incentives for response
  4. Use careful follow-up of non-respondents
  5. For online surveys, weight results to match known population characteristics

3Problem 3hard

❓ Question:

Compare three sources of bias in surveys: sampling error, undercoverage, and non-response. Give an example showing how all three could occur in one survey and explain which are most problematic for decision-making.

💡 Show Solution

Sampling Error:

What it is: Natural variability in sample statistics due to random sampling. Even with a perfectly unbiased sample, different samples give different results.

Example: Two SRS of 500 voters each might give 52% and 48% support for a candidate just by chance.

Is it a bias?: NO — it's unavoidable randomness, not systematic error. Reduced by larger sample sizes.

Undercoverage:

What it is: Some population groups are systematically excluded or harder to reach.

Example: Phone survey doesn't reach people without phones (young, mobile-only) or language minorities.

Result: Sample doesn't reflect population structure; biased estimates.

Is it a bias?: YES — systematic, not random.

Non-response:

What it is: People who don't respond differ systematically from those who do.

Example: Survey of job satisfaction has 30% response rate. Non-responders might be very satisfied (don't bother responding) or very dissatisfied (don't want to engage).

Result: Biased estimate depending on who non-responders are.

Is it a bias?: YES — systematic, not random.

Example: All three in one survey

A company conducts phone survey about workplace satisfaction:

  • Sampling error: They randomly select 300 employees, but by chance, younger workers (more critical) are overrepresented → slight downward bias in satisfaction estimate
  • Undercoverage: Survey doesn't reach part-time remote workers or recently hired employees without company phones → missing diverse perspectives
  • Non-response: 40% of called employees don't answer or decline; non-responders are either very satisfied (busy) or very dissatisfied (disengaged) → middle ground overrepresented

Result: Satisfaction estimate is biased AND has random variability.

Which biases are most problematic?

Ranking by severity:

  1. Non-response bias (most problematic) — directly distorts results; unknown magnitude
  2. Undercoverage (very problematic) — missing entire subgroups; results don't apply to all
  3. Sampling error (least problematic) — unavoidable but understood; reduced with larger samples; we can quantify it with confidence intervals

Why?: Sampling error is predictable and decreases with n. Biases are systematic distortions we can't easily quantify or correct without knowing the truth.

Lesson for decision-making: Always ask: "Who is missing?" (undercoverage) and "Who didn't respond?" (non-response). These are far more problematic than sampling error.

Explain using:

⚠️ Common Mistakes: Bias in Sampling and Surveys

Avoid these 3 frequent errors

📌 Related Topics in Unit 3: Collecting Data

❓ Frequently Asked Questions

What is Bias in Sampling and Surveys?▾
Identify sources of bias in sampling and surveys including voluntary response and convenience sampling.
How can I study Bias in Sampling and Surveys effectively?▾
Start by reading the study notes and working through the examples on this page. Then use the flashcards to test your recall. Practice with the 3 problems provided, checking solutions as you go. Regular review and active practice are key to retention.
Is this Bias in Sampling and Surveys study guide free?▾
Yes — all study notes, flashcards, and practice problems for Bias in Sampling and Surveys on Study Mondo are free to access. No account is needed.
What course covers Bias in Sampling and Surveys?▾
Bias in Sampling and Surveys is part of the AP Statistics course on Study Mondo, specifically in the Unit 3: Collecting Data section. You can explore the full course for more related topics and practice resources.
Are there practice problems for Bias in Sampling and Surveys?▾
Yes, this page includes 3 practice problems with detailed solutions. Each problem includes a step-by-step explanation to help you understand the approach.