analyze-prod-issue

Analyze Prod Issue

Take a production symptom (e.g. "parents can't book") and diagnose it end-to-end by correlating two sources — Honeycomb for the error shape, the prod DB error_tracker tables for the who + why — then scope impact honestly and optionally file a scrubbed issue.

Type: Rigid workflow. Follow steps exactly.

Prerequisite: read-only prod DB access must be set up (bin/prod-db). If it errors, see docs/runbooks/prod-db-access.md first.

The exact Honeycomb tool calls and prod SQL are in references/cookbook.md — read it when a step says to.

Step 1 — Orient (Honeycomb)

Call mcp__honeycomb__get_workspace_context. Note the environment slugs; production is the live environment (there is also dev). Use live in every subsequent Honeycomb call.

Done when: you know the live env slug and its dataset.

Step 2 — Find the error surface

• mcp__honeycomb__list_spans over 14d — ranks span names by count with root_count. Scan for the failing area and for funnel drops (an entry span with many mounts but its terminal action span near zero often is the reported symptom).
• mcp__honeycomb__run_query with filter error=true, breakdown ["name"], time_range "14d" — ranks the actually-failing spans by count.

Done when: you have candidate failing span name(s) with counts. If there are zero error=true spans, the symptom is not an exception — pivot to the funnel-drop reading.

Step 3 — Trace the causal chain

mcp__honeycomb__get_trace on a failing trace (show_events: true). Read the waterfall: find the span where the error is raised and the call chain above it (HTTP entry → worker → context function → repo query).

Honeycomb gap to expect: a DB error inside a rolled-back transaction surfaces only as query_error / status_message with no exception text (spans carry no event). That absence is the signal to go to the prod DB — do not stop here.

Done when: the failing chain and the raising span are mapped.

Step 4 — Pull the real reason (prod DB)

Query error_tracker via bin/prod-db -c "<SQL>" (read-only reader role). See references/cookbook.md for the schema and the exact queries.

• error_tracker_errors — kind, reason, source_function, last_occurrence_at, status. Find the error matching the trace.
• error_tracker_occurrences — context (Oban job.args + the full event payload: event_type, entity_id, user email/name), stacktrace, breadcrumbs, and job.state / job.attempt.

This is where the swallowed DB error, the affected user, the triggering event, and the Oban disposition (retryable vs discard) actually live.

Done when: you know the underlying error, the affected entity, the triggering event, and the occurrence count + date range.

Step 5 — Cross-reference schema & data

Confirm the hypothesis against real rows before believing it:

• Does the row the handler tried to create already exist? (duplicate vs genuinely missing)
• What are the actual constraint/index names? (a changeset unique_constraint name mismatch makes a violation raise instead of returning a changeset error)
• Affected-vs-total counts — how many entities are in the bad state vs the population?

Done when: the mechanism is proven from data, not assumed.

Step 6 — Scope impact honestly

The point of Step 5 is to correct the premise, not confirm it. Produce a one-paragraph impact statement that:

• Separates noise vs a real user-blocker — an erroring/discarded job is not automatically a blocked user (the side effect may already have succeeded on another path).
• Quantifies affected users by count.
• Splits disjoint phenomena — two symptoms with the same surface can have different causes and different populations; do not conflate them.

Done when: an honest, numeric impact statement exists — even if it contradicts the original complaint.

Step 7 — Confirm root cause in code

Grep the implicated modules to anchor the fix. State explicitly whether the bug exists on main or the behaviour is a live-vs-main deploy lag (live can raise what main already handles). Check the current commit on live if it matters.

Done when: the fix direction names concrete files and says whether deploying main fixes it.

Step 8 — (Optional) File a scrubbed issue

If the finding warrants tracking, hand off to /create-issue.

Public-repo PII rule (non-negotiable): reference affected users by count + opaque UUID only. Never put names or emails in a public issue — GitHub edit history is public, so scrubbing after the fact requires deleting and recreating the issue. Keep the user_id ↔ identity mapping in the DB, not the issue.

Done when: the issue exists with count-only references, or you and the user decide not to file.

Rules

• Two sources, not one. Honeycomb gives the shape (chain, span, timing); error_tracker gives the who + why (reason, event payload, stacktrace, Oban state). Neither alone is enough.
• Read-only always. Reach prod only through bin/prod-db (the SELECT-only reader role). Never open a write path against production.
• Count, never names, in public artifacts. PII stays in the DB; issues and PRs get counts and UUIDs.
• Scope honestly. "Erroring" ≠ "user-blocked." Prove impact from data before claiming it, and correct the premise when the data disagrees.
• The rollback hides the error. When Honeycomb shows only :rollback / query_error with no text, the real reason is in error_tracker_occurrences.context, not the span.