mutation-testing

Mutation Testing

Add a third validation layer to Outside-In TDD workflow. Acceptance tests verify WHAT (observable behavior), Domain tests verify HOW (business rules), mutation testing verifies tests actually catch bugs.

Core Concept

Mutation testing introduces deliberate bugs (mutants) into source code, then runs the test suite. If tests fail, the mutant is killed ✓. If tests pass despite the bug, the mutant survives ✗ (test gap found).

Source code → introduce mutation → run tests
                                     ├── tests FAIL → mutant killed ✓
                                     └── tests PASS → mutant survived ✗

A project with 100% code coverage can still have a 60% mutation score — meaning 40% of introduced bugs go undetected.

When to Use

Run mutation testing after the relevant test baseline is green:

1. ✅ Core behavior tests pass
2. ✅ Rule-focused tests pass
3. 🧬 Mutation testing — verify tests detect regressions

Never run on red baseline — mutation assumes tests work correctly first.

Approach for .NET/C#

Primary: Stryker.NET (Recommended)

For .NET projects, Stryker.NET is the established mutation framework with excellent C# support.

No stryker-config.json — by design. This project does NOT use a Stryker config file. Never create one, and never run bare dotnet stryker (it relies on a config or mutates everything). Always pass --project, -tp, and --since explicitly so the run is reproducible and scoped to the diff.

Install (only if not already available):

# Check first — if this succeeds, skip installation entirely. Do NOT manipulate PATH.
dotnet stryker --version

# Only run if the above command fails (tool not found)
dotnet tool install -g dotnet-stryker

Primary command — diff vs the destination branch, no config file:

# Mutate the Domain project, run its UnitTests, scope to the PR diff.
# Replace <target-branch> with the branch you will merge INTO (the PR base).
dotnet stryker \
  --project YourProject.Domain.csproj \
  -tp tests/YourProject.UnitTests/YourProject.UnitTests.csproj \
  --since:<target-branch> \
  -r markdown -r json -r cleartext

Why --since: it mutates only the code changed between the current branch and <target-branch> (the PR base) — fast, focused on what the PR actually touches. In CI, resolve the base from the PR event and prefix with the remote if needed (e.g. --since:origin/main). If the diff touches no Domain code, Stryker reports zero mutants — that is a valid PASS, not a failure.

--project takes the .csproj FILE NAME, not a path — Stryker locates it inside the solution. The no-config form is --project + -tp + --since, with NO --mutate path globs.

⚠️ Footgun — project-relative --mutate: --mutate globs are resolved RELATIVE TO THE MUTATED PROJECT directory, not the solution root. A pattern like --mutate "src/YourProject.Domain/**/*.cs" matches NOTHING from inside the Domain project, so every mutant is silently "Removed by mutate filter" and Stryker reports "unable to calculate a mutation score". To exclude files, use suffix-only EXCLUDE patterns (--mutate "!**/*Marker.cs"); never prefix an include glob with a solution-relative path.

Exclude non-logic files (combine with --since, suffix-only patterns are safe):

dotnet stryker \
  --project YourProject.Domain.csproj \
  -tp tests/YourProject.UnitTests/YourProject.UnitTests.csproj \
  --since:<target-branch> \
  --mutate "!**/*Marker.cs" --mutate "!**/DependencyInjection.cs" \
  -r markdown -r json -r cleartext

Cumulative baseline — full picture across PRs:

# --with-baseline = --since + a persistent baseline report. Keeps a full score
# history in CI while only re-testing code changed vs the target branch.
dotnet stryker \
  --project YourProject.Domain.csproj \
  -tp tests/YourProject.UnitTests/YourProject.UnitTests.csproj \
  --with-baseline:<target-branch> \
  -r markdown -r json

Alternative: Custom Mutation Tool (For Specific Needs)

Build a custom tool only when:

• Stryker doesn't cover domain-specific mutation patterns
• You need tight integration with custom test infrastructure
• Performance optimization requires targeted mutation scope

Architecture (3 modules):

1. Mutations — rules table (+ → -, true → false, >= → >)
2. Runner — source-to-test mapping, targeted test execution
3. Core — orchestration: apply mutation → run tests → restore → report

For full custom tool reference, see Uncle Bob's empire-2025 mutation testing.

Core Mutation Categories

Category	Examples
Arithmetic	+ ↔ -, * ↔ /, ++ ↔ --
Comparison	> ↔ >=, < ↔ <=, == ↔ !=
Boolean	true ↔ false, && ↔ ||, !x ↔ x
Conditional	negate conditions, swap if/else branches
Constant	0 ↔ 1, "" ↔ "mutant", null ↔ new()
Return value	return true → return false
Void method	remove method call entirely
LINQ	.Any() ↔ .All(), .First() ↔ .Last()

Workflow

Universal prerequisite — applies to every step, every scenario: Before any mutation activity (first run, CI setup, killing survivors, analyzing reports), the test suite for the affected scope must be green. If tests are failing, fix them first. Mutation results on a red baseline are meaningless — failing tests cannot kill mutants they already can't run.

Step 1: Verify Prerequisites

Before running mutation testing, confirm:

• ✅ Baseline tests are green for the mutated scope
• ✅ Meaningful unit tests exist (mutation runs against unit tests)
• ✅ No uncommitted changes (mutations modify source temporarily)
• ✅ Tests are fast (< 100ms each) — slow tests = slow mutation runs

Step 2: Set Mutation Scope

Target critical business logic first:

• Domain policies, decision engines, pricing/risk calculators
• Application orchestration with complex conditionals
• Validation rules and boundary behavior

Exclude from mutation:

• DTOs, data structures without logic
• Infrastructure (repositories, adapters)
• Configuration, DependencyInjection files
• Generated code, marker interfaces

Progressive scoping:

Phase	Scope	Goal
Week 1-2	One critical rule module	Baseline + learning
Week 3-4	All core rule modules	Establish quality gate
Ongoing	Core + critical orchestration handlers	Full confidence

Step 3: Run Mutations

Scope to the PR diff (default — no config file, compare vs the target branch):

dotnet stryker \
  --project YourProject.Domain.csproj \
  -tp tests/YourProject.UnitTests/YourProject.UnitTests.csproj \
  --since:<target-branch> \
  -r markdown -r json -r cleartext

<target-branch> is the PR base (the branch you merge INTO). In CI, resolve it from the PR event and prefix with the remote if needed (--since:origin/main).

Metrics:

• Total mutants generated
• Mutants killed (tests caught the bug ✓)
• Mutants survived (test gap ✗)
• Mutation score: (killed / total) × 100

--since note: unchanged files produce no result — this is expected. Survivors and kills only apply to the diff scope. A PR with no Domain changes yields zero mutants (valid PASS).

Expected duration: --since run: ~1-3 min on the changed scope.

Step 4: Analyze Survivors

Query survivors directly from the JSON report — do not read the full file:

jq '[.files | to_entries[] | {file: .key, survivors: [.value.mutants[] | select(.status == "Survived") | {mutator: .mutatorName, line: .location.start.line, replacement: .replacement}]}] | map(select(.survivors | length > 0))' \
  StrykerOutput/$(ls -t StrykerOutput | head -1)/reports/mutation-report.json

For each surviving mutant:

1. Read the mutation — what was changed? (e.g., >= → >, removed if branch)
2. Identify unguarded behavior — which business rule isn't tested?
3. Categorize:
   • Real gap — behavior change not caught by tests
   • Equivalent mutant — mutation doesn't change observable behavior

Equivalent mutant examples:

• x = x + 0 changed to x = x + 1 (dead code)
• Logging statements removed (no observable effect)
• Defensive null checks when value is guaranteed non-null by type

After classifying survivors, always include a targeted re-run command scoped to the files that contain real gaps — this confirms kills after you write new tests and gives reviewers a runnable artifact:

dotnet stryker \
  --project YourProject.Domain.csproj \
  -tp tests/YourProject.UnitTests/YourProject.UnitTests.csproj \
  --mutate "**/<FileWithRealGap>.cs" \
  --mutate "!**/*Marker.cs" --mutate "!**/DependencyInjection.cs" \
  -r cleartext

Step 5: Kill Surviving Mutants

For each real survivor (not equivalent):

1. Write a new test targeting the unguarded behavior
2. Run test against mutated code (using Stryker's mutation operator):
   • Expected: test FAILS (catches the bug)
3. Run test against original code:
   • Expected: test PASSES
4. Re-run Stryker to confirm kill

Example:

Survivor: if (age >= 18) mutated to if (age > 18) → survived

// New test to kill the boundary mutant
[Fact]
public void WhenDriverIsExactly18_ShouldBeEligible()
{
    var policy = new EligibilityPolicy();
    var driver = new DriverInfo(Age: 18, LicenseYears: 1);
    var vehicle = new VehicleInfo(Type: "sedan", Age: 1);

    var result = policy.Evaluate(driver, vehicle);

    Assert.True(result.IsEligible); // Fails if mutant uses `age > 18`
}

Step 6: Report & Document

Present summary with before/after metrics:

Mutation Testing Report — Core Business Layer
══

---

*Content truncated.*