Bias in Sampling and Surveys
Identify sources of bias in sampling and surveys including voluntary response and convenience sampling.
Try the Interactive Version!
Learn step-by-step with practice exercises built right in.
⚠️ Bias in Sampling and Surveys
Types of Sampling Bias
Undercoverage
- Some population members systematically excluded
- Sampling frame does not include all population members
- Example: telephone survey misses homeless, institutionalized, no-phone individuals
- Results biased toward accessible subgroup
Nonresponse Bias
- People who don't respond differ from responders
- Higher rates in mail surveys (~50%), lower in in-person (~80%)
- Cannot be completely eliminated, can be reduced via multiple follow-ups
- Example: survey on financial habits—wealthy more likely to refuse
Voluntary Response Bias
- Self-selected responders not representative
- Highly motivated to respond (strongly agree or disagree)
- Example: online polls, call-in surveys
- Results typically more extreme than population opinion
Types of Response Bias
Wording Effects
- Question phrasing influences answer
- Example: "Should government spend more on welfare?" vs "Should government spend more on helping poor?"
- Leading questions subtly suggest preferred answer
- Double negatives confuse respondents
Interviewer Effects
- Interviewer appearance, tone, demographics influence response
- Subjects give socially desirable answers (social desirability bias)
- Example: respondent gives less honest answer to attractive interviewer
- Reduced by trained, neutral interviewer
Question Order Effects
- Previous questions prime respondents
- Example: ask about healthcare costs, then satisfaction → answers affected
- Randomize or pilot-test question order
Timing and Setting
- When/where survey taken affects response
- Example: customer satisfaction survey in store vs. email (both environments bias)
Strategies to Reduce Bias
| Bias | Reduction |
|---|---|
| Undercoverage | Use comprehensive sampling frame; multistage or stratified to reach hard-to-reach groups |
| Nonresponse | Multiple contact attempts, incentives, follow-up with non-respondents |
| Voluntary response | Use random sampling instead |
| Wording | Pilot-test questions; use neutral language; avoid leading questions |
| Interviewer | Train interviewers; use telephone/email; randomize interviewer assignment |
| Social desirability | Ensure anonymity; use indirect questions (third-party phrasing) |
Common Mistakes
- Assuming larger sample size eliminates bias (bias is fixed-size problem, not variability)
- Confusing bias (systematic) with variability (random)
- Believing no-response survey can be unbiased without follow-up
Decision Rule
Is bias fixed (doesn't decrease with larger )? → Reduce via improved design Is variability decreasing with ? → Increase sample size
AP Exam Tip
When critiquing a survey, identify: potential source of bias (which type), explain mechanism, and propose mitigation strategy. Larger samples fix variability, not bias.
📚 Practice Problems
1Problem 1easy
❓ Question:
A survey asking 'Don't you agree that more funding should go to schools?' is an example of what type of bias? Explain what response bias is and how to fix it.
💡 Show Solution
This is an example of response bias (specifically leading question bias or wording bias).
What response bias means: Response bias occurs when question wording, question order, or surveyor behavior influences responses, causing answers to differ from true opinions.
Why this question is biased: The phrasing 'Don't you agree...' is leading — it suggests the desired answer is 'yes.' Most respondents will comply rather than disagree with a leading question, inflating support for school funding.
How to fix it:
Reword as neutral and balanced alternatives:
- 'How should school funding change?' (open-ended)
- 'Do you support increasing, maintaining, or decreasing school funding?' (balanced options)
- 'School funding should be increased' vs. 'School funding should be maintained' (state both sides)
Key principle: Survey questions must be neutral and not suggest a particular answer. Both sides of an issue should be presented equally.
Other response bias types: Acquiescence bias (always agreeing), social desirability bias (answering what sounds good, not true), and question order effects.
2Problem 2medium
❓ Question:
An online poll asks: 'Have you voted in the last election?' Those who don't respond are removed from the survey. What biases might this introduce?
💡 Show Solution
Biases introduced:
1. Nonresponse bias:
People who don't respond differ from respondents. Who doesn't respond?
- Busy people (less time to fill out surveys)
- Less interested in voting/politics
- Elderly or tech-unfamiliar people (online survey may exclude them)
- People with contrary opinions (may ignore it)
Result: Sample overrepresents voters and politically engaged people; underrepresents non-voters.
2. Voluntary response bias (self-selection):
Online polls are voluntary. Who volunteers?
- People with strong opinions (both for and against)
- More motivated individuals
- Those with internet access
Result: Response represents passionate people, not average population.
3. Undercoverage (selection bias):
Only people with internet access can take online survey. Missing:
- Elderly, rural poor (limited internet)
- Low-income households
- Tech-averse individuals
These groups have different voting patterns than internet users.
4. If the question assumes voting:
'Have you voted...?' assumes respondents are voters. People who think they shouldn't answer (non-voters, non-citizens) might skip the survey, creating further nonresponse bias.
Consequences:
Estimated voter turnout would be inflated — the poll would overestimate how many people actually voted.
How to reduce bias:
- Use random sampling (not voluntary signup)
- Use multiple data-collection methods (phone, mail, in-person) to reach diverse groups
- Offer incentives for response
- Use careful follow-up of non-respondents
- For online surveys, weight results to match known population characteristics
3Problem 3hard
❓ Question:
Compare three sources of bias in surveys: sampling error, undercoverage, and non-response. Give an example showing how all three could occur in one survey and explain which are most problematic for decision-making.
💡 Show Solution
Sampling Error:
What it is: Natural variability in sample statistics due to random sampling. Even with a perfectly unbiased sample, different samples give different results.
Example: Two SRS of 500 voters each might give 52% and 48% support for a candidate just by chance.
Is it a bias?: NO — it's unavoidable randomness, not systematic error. Reduced by larger sample sizes.
Undercoverage:
What it is: Some population groups are systematically excluded or harder to reach.
Example: Phone survey doesn't reach people without phones (young, mobile-only) or language minorities.
Result: Sample doesn't reflect population structure; biased estimates.
Is it a bias?: YES — systematic, not random.
Non-response:
What it is: People who don't respond differ systematically from those who do.
Example: Survey of job satisfaction has 30% response rate. Non-responders might be very satisfied (don't bother responding) or very dissatisfied (don't want to engage).
Result: Biased estimate depending on who non-responders are.
Is it a bias?: YES — systematic, not random.
Example: All three in one survey
A company conducts phone survey about workplace satisfaction:
- Sampling error: They randomly select 300 employees, but by chance, younger workers (more critical) are overrepresented → slight downward bias in satisfaction estimate
- Undercoverage: Survey doesn't reach part-time remote workers or recently hired employees without company phones → missing diverse perspectives
- Non-response: 40% of called employees don't answer or decline; non-responders are either very satisfied (busy) or very dissatisfied (disengaged) → middle ground overrepresented
Result: Satisfaction estimate is biased AND has random variability.
Which biases are most problematic?
Ranking by severity:
- Non-response bias (most problematic) — directly distorts results; unknown magnitude
- Undercoverage (very problematic) — missing entire subgroups; results don't apply to all
- Sampling error (least problematic) — unavoidable but understood; reduced with larger samples; we can quantify it with confidence intervals
Why?: Sampling error is predictable and decreases with n. Biases are systematic distortions we can't easily quantify or correct without knowing the truth.
Lesson for decision-making: Always ask: "Who is missing?" (undercoverage) and "Who didn't respond?" (non-response). These are far more problematic than sampling error.
⚠️ Common Mistakes: Bias in Sampling and Surveys
Avoid these 3 frequent errors
Practice with Flashcards
Rate this topic's cards with spaced repetition. Cards join your deck when you finish a topic's lesson and take its exit quiz.
Browse All Topics
Explore more AP Statistics topics