alterlab-mixed-methods

Mixed Methods Research Design

Overview

Mixed methods research is a methodology that combines quantitative and qualitative research approaches within a single study or program of inquiry. It goes beyond simply collecting both types of data — it requires intentional integration at one or more stages of the research process (design, methods, interpretation, reporting) to generate insights that neither approach could produce alone. This skill provides comprehensive guidance on designing, executing, and reporting mixed methods studies according to established frameworks, with particular emphasis on the Creswell & Plano Clark typology and the integration strategies that distinguish rigorous mixed methods from mere parallel data collection.

Mixed methods emerged as a recognized "third methodological movement" in the early 2000s, though researchers had been combining approaches for decades. Its legitimacy rests on the philosophical position of pragmatism — selecting methods based on what works best to answer the research questions rather than adhering to a single paradigm. Today, mixed methods is a requirement or strong recommendation in many funding agencies (NIH, NSF, ESRC) and is increasingly expected in fields such as health sciences, education, evaluation research, and social policy.

When to Use This Skill

Use this skill when:

• Designing a study that requires both statistical generalizability and contextual depth
• Planning how to combine qualitative and quantitative strands in a coherent design
• Choosing between convergent, explanatory sequential, exploratory sequential, embedded, transformative, or multiphase designs
• Developing integration strategies (merging, connecting, embedding) for multi-strand studies
• Constructing joint displays to visualize integrated findings
• Formulating meta-inferences that go beyond what either strand alone could produce
• Writing the methods section of a mixed methods manuscript or proposal
• Evaluating the quality, legitimacy, or rigor of a mixed methods study
• Using the Creswell & Plano Clark notation system to diagram a design
• Selecting software tools for managing mixed methods data integration
• Preparing a mixed methods study for IRB review with multiple data collection phases
• Responding to reviewer critiques about the rationale for mixing methods

Core Capabilities

1. Mixed Methods Design Typology

The following designs represent the major archetypes. In practice, many studies adapt or combine these.

Convergent Parallel Design (QUAL + QUAN)

Both strands are collected and analyzed concurrently, then merged for interpretation.

Notation: QUAL + QUAN → Merge → Interpretation

Timeline:
  Phase 1 (concurrent):
    ├── Quantitative data collection & analysis
    └── Qualitative data collection & analysis
  Phase 2:
    └── Merge results → Compare, contrast, synthesize

Purpose: Triangulation, complementarity, or obtaining a more complete understanding

Example research question:
  "How do survey measures of teacher self-efficacy converge with or diverge
   from teachers' narrative accounts of their classroom confidence?"

Key decisions in convergent design:

• Sample: Same participants (identical samples), overlapping samples, or parallel samples?
• Timing: Truly concurrent or within the same broad phase?
• Merging point: At the results level (side-by-side comparison) or at the data transformation level (qualitizing or quantitizing)?
• Handling discrepancy: What if findings conflict? Plan for this a priori.

A worked convergent-design joint display (survey statistics vs. interview themes with per-row convergence assessment and meta-inference) is in references/joint_display_templates.md.

Explanatory Sequential Design (QUAN → qual)

Quantitative data is collected and analyzed first; qualitative data is then collected to explain or elaborate on quantitative results.

Notation: QUAN → qual → Interpretation

Timeline:
  Phase 1: Quantitative data collection & analysis
  Phase 2: Identify results needing explanation
           (outliers, unexpected findings, significant predictors)
  Phase 3: Qualitative data collection targeting those results
  Phase 4: Qualitative analysis
  Phase 5: Integration and interpretation

Connection point: Quantitative results inform qualitative sampling and protocols

Example research question:
  "What factors explain why some departments showed significant improvement
   in research output while others with similar resources did not?"

Participant selection strategies for the qualitative phase:

# Conceptual selection logic for explanatory sequential design
def select_qualitative_participants(quant_results):
    """
    Common strategies for selecting participants
    for the qualitative follow-up phase.
    """
    strategies = {
        "extreme_cases": "Select participants at the extremes of the distribution",
        "outlier_cases": "Select participants whose outcomes deviate from predictions",
        "typical_cases": "Select participants near the mean for each subgroup",
        "maximal_variation": "Select across the full range of the key variable",
        "confirming_disconfirming": "Select cases that support and challenge quant findings",
        "subgroup_follow_up": "Select from each statistically identified subgroup"
    }

    # Example: Follow up on regression residuals
    high_residual = quant_results[quant_results['residual'].abs() > 2.0]
    typical = quant_results[quant_results['residual'].abs() < 0.5].sample(n=5)

    return {
        "outliers": high_residual,
        "typical": typical,
        "rationale": "Comparing outlier and typical cases to explain model misfit"
    }

Exploratory Sequential Design (qual → QUAN)

Qualitative data is collected first to explore a phenomenon; findings inform the development of a quantitative instrument or phase.

Notation: qual → QUAN → Interpretation

Timeline:
  Phase 1: Qualitative data collection & analysis
  Phase 2: Develop quantitative instrument/variables from qual findings
           (e.g., survey items derived from interview themes)
  Phase 3: Quantitative data collection & analysis
  Phase 4: Integration and interpretation

Connection point: Qualitative findings generate hypotheses, variables, or instruments
                  for the quantitative phase

Common applications:
  - Instrument development (qual themes → survey items → psychometric validation)
  - Taxonomy development (qual categories → quantitative classification testing)
  - Theory generation (qual grounded theory → quantitative hypothesis testing)

Instrument development workflow:

Qualitative Phase:
  Interviews (n=20-30) → Thematic analysis → Identify constructs and language

Connecting Bridge:
  Themes → Item pool → Expert review → Cognitive interviews → Pilot items

Quantitative Phase:
  Survey (n=300+) → Exploratory factor analysis → Confirmatory factor analysis
  → Reliability testing → Validity evidence (convergent, discriminant, criterion)

Embedded Design (QUAN[qual] or QUAL[quan])

One strand is primary; the other is embedded within it to enhance the primary design.

Notation: QUAN(qual) — Quantitative primary with embedded qualitative

Example: Randomized controlled trial with embedded qualitative process evaluation

  ┌─────────────────────────────────────────────┐
  │  RCT (Primary Quantitative Design)          │
  │                                              │
  │  Treatment group ──→ Outcome measures        │
  │       │                                      │
  │       └──→ [Qualitative interviews at        │
  │             midpoint to understand            │
  │             implementation fidelity           │
  │             and participant experience]       │
  │                                              │
  │  Control group ──→ Outcome measures          │
  └─────────────────────────────────────────────┘

Purpose: The qualitative strand answers a secondary question
         within the larger quantitative framework

Transformative Design

Any mixed methods design organized within a transformative theoretical framework (feminist, critical race theory, disability studies, postcolonial, etc.) that centers equity, justice, and the perspectives of marginalized communities.

Key principles:
  1. Research questions address power, oppression, or social justice
  2. Marginalized community members participate in design decisions
  3. Methods are selected to amplify silenced voices
  4. Integration explicitly examines how findings relate to structural inequity
  5. Results include action agendas and recommendations for change

Notation adds a framework wrapper:
  Transformative Framework [QUAL → QUAN]
  or
  Transformative Framework [QUAL + QUAN]

Multiphase Design

A programmatic approach where multiple mixed methods projects build on each other over time, common in large-scale program evaluation and longitudinal research.

Notation: Study 1 (QUAN) → Study 2 (qual) → Study 3 (QUAL + QUAN) → ...

Example: Multi-year curriculum evaluation program
  Year 1: Needs assessment (QUAL → QUAN)
  Year 2: Pilot intervention (QUAN with embedded qual)
  Year 3: Full-scale RCT (QUAN → qual for process evaluation)
  Year 4: Sustainability study (QUAL + QUAN convergent)

2. The Notation System

The Creswell & Plano Clark notation system communicates design decisions concisely:

Symbol Reference:
  UPPERCASE   = Priority/emphasis strand (e.g., QUAN = quantitative is primary)
  lowercase   = Secondary/supplementary strand (e.g., qual = qualitative is secondary)
  +           = Concurrent/simultaneous collection
  →           = Sequential collection (left happens before right)
  ( )         = Embedded strand within a larger design
  [ ]         = Framework wrapper (e.g., transformative, pragmatic)
  { }         = Sometimes used for the integration/merging phase

Common patter

---

*Content truncated.*