Research Methods & Study Design

Research Methods & Study Design

Part 1 of 4 — Variables, Sampling & Study Types

Types of Variables

Sampling Methods

Study Types (by Causation Inference)

Key: Only random assignment (RCT) can establish causation by balancing confounds.

Variables & Sampling 🎯

Key Takeaways — Part 1

• IV = Independent Variable (what's manipulated); DV = Outcome (what's measured)
• Confound = Uncontrolled variable related to both the IV and the DV
• Random Assignment ⟹ RCT ⟹ Strongest causation inference (internal validity)
• Random Sampling ⟹ Representative sample ⟹ Generalizability (external validity); convenience sampling weakens it
• Sampling decides WHO is studied; assignment decides WHICH GROUP they join — only assignment supports causation
• Matching (matched pairs) is a design control: it controls specific confounds but not unknown ones (inferior to randomization)

Worked Examples — Variables, Sampling & Study Types

Question: Participants receive either 0 mg, 50 mg, or 100 mg caffeine, then complete a reaction-time test.

1. Manipulated factor is caffeine dose.
2. Measured outcome is reaction time.

IV: caffeine dose. DV: reaction time.

Question: A stress survey recruits only pre-med students from one campus.

1. Recruitment is convenience-based and narrow.
2. Sample likely differs from general population.

Main issue: selection/sampling bias, reducing external validity.

Question: Researchers record average sleep and exam scores without assigning sleep schedules.

1. No manipulation of sleep duration.
2. No random assignment.

Design type: observational/correlational, not experimental.

Research Methods & Study Design

Part 2 of 4 — Validity & Threats to Validity

Types of Validity

Threats to Internal Validity

Threats to External Validity

Solution: Random assignment (internal validity) + Representative (random) sample (external validity)

Validity & Threats 🎯

Key Takeaways — Part 2

• Internal Validity: Random assignment controls confounds, so a change in the DV can be attributed to the IV
• External Validity: Representative sampling + diverse conditions enable generalization
• Construct Validity: Measurement must truly measure the construct intended
• Key Threats: Confounding, selection bias, attrition (internal); volunteer bias, lab conditions (external)
• Ideal Study: Random assignment (strong internal) + Representative (random) sample (strong external)

Worked Examples — Validity & Threats

Question: A tightly controlled inpatient trial shows strong results, but participants are all healthy young adults.

1. Strong protocol control supports internal validity.
2. Narrow participant profile limits generalization.

Result: high internal validity, weaker external validity.

Question: In a weight-loss trial, dropout is 35% in treatment vs 8% in control.

1. Unequal dropout changes who remains in each group.
2. Final comparison can be biased.

Threat: attrition bias.

Question: "Empathy" is measured only by number of social media comments posted.

1. Metric may not map to the intended construct.
2. Behavior proxy is weak and noisy.

Main concern: construct validity.

Research Methods & Study Design

Part 3 of 4 — Blinding & Experimental Controls

Blinding in Studies

Mechanism of Placebo Effect: Brain expectation → neurotransmitter release → real physiological changes. Blinding does not abolish the placebo effect; it makes it equal in both arms so a placebo control can subtract it.

• Typical placebo response rates are often quoted at roughly 30-40% of patients, though the figure varies widely by condition and outcome.

Placebo & Control Groups

Example: Depression RCT

• Placebo group: 30% improve (placebo effect + natural recovery)
• Antidepressant group: 60% improve
• True drug effect = 60% − 30% = 30%

Types of Experimental Designs

Carryover Effect: Practice in Condition A affects performance in Condition B. Counterbalance (or randomize) the order, or use a between-subjects design.

Blinding & Controls 🎯

Key Takeaways — Part 3

• Double-Blind > Single-Blind > No Blinding (equalizes expectancy effects across groups + minimizes researcher bias)
• Placebo Control isolates IV effect by subtracting baseline placebo response
• Within-Subjects design increases power (subjects are own controls); risk of carryover effects
• Between-Subjects design avoids carryover; requires larger N
• MCAT Tip: Always check: Is the study blinded? Is there a placebo control? Could carryover effects bias results?

Worked Examples — Blinding & Experimental Controls

Question: In a pain trial, nurses who know treatment assignment rate pain outcomes more favorably for the drug group.

1. Knowledge of assignment can influence ratings.
2. Blinding raters removes expectancy effects.

Fix: use a double-blind design.

Question: Placebo improves 25%; drug improves 55%.

1. Total observed drug-group response includes placebo component.
2. Subtract placebo baseline.

Estimated drug-specific effect: 30 percentage points.

Question: Subjects take memory test after caffeine in week 1 and after placebo in week 2. Scores improve in week 2.

1. Practice effects can increase later scores independent of treatment.
2. Order effects threaten interpretation.
3. Counterbalancing or between-subjects design can reduce this issue.

Threat identified: carryover/order effect.

Research Methods & Study Design

Part 4 of 4 — Sample Size, Ethics & Meta-Analysis

Sample Size & Power

Larger sample → More power to detect true effects (↓β, Type II error)

Rule of thumb: Aim for 80%+ power (β ≤ 20%).

Research Ethics (MCAT focus: Belmont Principles, Informed Consent, IRB)

Special Populations: Extra protections for children, prisoners, and the cognitively impaired (consent may be limited or open to coercion; children give assent alongside parental permission)

Vulnerable Populations: Cannot be excluded from research solely for "protection"; must justify any exclusion

Meta-Analysis

Combines data from multiple studies to increase statistical power.

Advantage: Large sample → more generalizable conclusions
Risk: Publication bias (positive findings more likely to be published)

Forest Plot Interpretation: Vertical line at 1.0 (OR) means no effect; if the pooled confidence interval crosses it, the effect is not significant overall.

Sample Size, Ethics & Meta-Analysis 🎯

Key Takeaways — Part 4

• Sample Size: Larger N → More power (can detect small true effects)
• Power Goal: 80%+ (Type II error rate β ≤ 20%)
• Belmont Principles: Respect for persons (informed consent), beneficence, justice
• Informed Consent: Disclosure of risks/benefits; voluntary; ability to withdraw
• Vulnerable Populations: Include with extra protections, don't exclude
• Meta-Analysis: Combines studies for power, but publication bias can overestimate effects
• Forest Plots: If CI crosses 1.0 (OR), effect is not statistically significant
• MCAT Tip: Ethics questions focus on consent, IRB review, protection of vulnerable groups

Worked Examples — Power, Ethics & Meta-Analysis

Question: Trial with n=40 per arm yields p=0.09 for a moderate effect.

1. Small sample can underpower the test.
2. Increasing n reduces standard error.
3. Better power improves chance of detecting a true effect.

Most direct improvement: larger sample size.

Question: A participant signs a form but was never told key side effects.

1. Signature alone is insufficient.
2. Valid consent requires meaningful disclosure and understanding.

Ethics issue: informed consent was inadequate.

Question: Pooled odds ratio is 0.85 with 95% CI 0.62 to 1.10.

1. For OR, null value is 1.0.
2. CI crosses 1.0.
3. Pooled effect is not statistically significant.

Interpretation: possible benefit, but inconclusive overall.