analyze-failures

Playwright E2E Test Failure Analysis

Analyze Playwright test failures and delegate fixes to specialized subagents.

Input

Report location: $ARGUMENTS

Supported input formats:

1. GitHub Actions run URL (easiest)

   • Example: https://github.com/saleor/saleor-dashboard/actions/runs/21513974962
   • Will download merged-blob-reports artifact automatically

2. GitHub PR URL

   • Example: https://github.com/saleor/saleor-dashboard/pull/6292
   • Will find the latest failed CI run and download artifact

3. ZIP file - Downloaded artifact from CI

   • Example: ~/Downloads/merged-blob-reports.zip
   • Will extract, merge shards, and parse automatically

4. Folder with blob reports - Already extracted CI artifact

   • Contains: report-e2e-*.zip shard files
   • Will run npx playwright merge-reports and parse

5. Pre-merged JSON file - Already merged report

   • Example: ./merged-report.json
   • Uses directly

Handling GitHub URLs

If the input is a GitHub URL, download the artifact first:

For Actions run URL:

# Extract run ID from URL
RUN_ID="21513974962"  # from https://github.com/.../actions/runs/21513974962

# List artifacts to find merged-blob-reports
gh run view $RUN_ID --json artifacts

# Download the artifact
gh run download $RUN_ID -n merged-blob-reports -D ./playwright-failures/downloaded

For PR URL:

# Get the PR number
PR_NUM="6292"  # from https://github.com/.../pull/6292

# Get the LATEST run (most recent) for this PR's branch
# Use --limit 1 to get only the latest
RUN_INFO=$(gh run list --branch $(gh pr view $PR_NUM --json headRefName -q .headRefName) --limit 1 --json databaseId,status,conclusion,name)

# Check if any runs exist
if [ "$(echo $RUN_INFO | jq 'length')" -eq 0 ]; then
    echo "No CI runs found for this PR"
    # Ask user if they want to trigger tests (see below)
fi

# Get the run ID from the latest run
RUN_ID=$(echo $RUN_INFO | jq -r '.[0].databaseId')

# Download from the latest run
gh run download $RUN_ID -n merged-blob-reports -D ./playwright-failures/downloaded

If no runs exist on the PR:

Use AskUserQuestion to ask the user:

No CI runs found for PR #[NUMBER].

Would you like to trigger E2E tests? I can add a label to the PR:
- "run pw-e2e" - Triggers Playwright E2E tests
- "test deployment" - Triggers deployment + tests

Or you can wait for an existing run to complete.

If user agrees, add the label:

gh pr edit $PR_NUM --add-label "run pw-e2e"

Then stop and ask user to come back after test ends run.

--

After downloading, the artifact folder becomes the input for the prepare script.

Phase 0: Check Previous Attempts (Main Agent)

Before analyzing new failures, check if we've tried to fix these tests before.

Check Git History for Recent Test Fixes

# Look for recent commits that modified Playwright tests
git log --oneline -20 --all -- "playwright/tests/*.spec.ts" "playwright/pages/*.ts"

# Check if any commits mention the failing test IDs (e.g., SALEOR_124)
git log --oneline -10 --grep="SALEOR_124" --grep="fix" --all-match

Check Previous Attempts File

The skill stores previous fix attempts in ./playwright-failures/previous-attempts.json:

{
  "attempts": [
    {
      "date": "2024-01-30T10:00:00Z",
      "runId": "21513974962",
      "testId": "SALEOR_124",
      "file": "attributes.spec.ts",
      "diagnosis": "Selector timing issue - attributesRows not waiting",
      "fix": "Changed .count().toEqual() to .toHaveCount()",
      "result": "unknown",
      "commits": ["abc1234"]
    }
  ]
}

If Previous Attempts Exist

1. Read the file to see what was tried
2. Check if same tests are failing again - if yes, previous fix didn't work
3. Inform subagents about what was already tried so they don't repeat it
4. Update the result field if we now know the outcome

Example prompt addition for subagents:

## Previous Attempts (DON'T REPEAT THESE)

This test was fixed before but is failing again:
- Previous diagnosis: [from file]
- Previous fix: [from file]
- Commit: [hash]

The previous fix didn't work. Try a DIFFERENT approach.

Save New Attempts

After making fixes, update the file:

# The skill should append new attempts to the JSON file

Phase 1: Prepare Report (Main Agent)

⚠️ MANDATORY WORKFLOW - READ THIS FIRST

The main agent (you) MUST follow this workflow:

1. ✅ Download/prepare report files (Step 1.1)
2. ✅ Read summary.md and failures-full.json (Step 1.2)
3. ✅ Read screenshots AND error-context files (Step 1.2.1)
4. ⚠️ SPAWN Explore agents using Task tool (Step 1.3) - DO NOT skip this!
5. ✅ Wait for agents to return with results
6. ✅ Analyze results to determine test bug vs app bug (Step 1.4-1.6)
7. ✅ Only THEN proceed to fix tests if needed (Phase 2)

YOU MUST NOT:

• Run git log, git diff, or explore code directly
• Skip spawning Explore agents
• Proceed to fixing before exploration completes

---

Step 1.1: Handle Input & Download if Needed

Check for Existing Report Directory

Before downloading, check if ./playwright-failures/ already exists:

if [ -d "./playwright-failures" ]; then
    # Directory exists - ask user what to do
fi

If directory exists, use AskUserQuestion:

A previous playwright-failures directory exists.

How would you like to proceed?
1. Delete the old report and download fresh
2. Keep the old report (rename to playwright-failures-[timestamp]) and download fresh
3. Use the existing report (skip download)

Based on user choice:

# Option 1: Delete old
rm -rf ./playwright-failures

# Option 2: Keep old with timestamp
TIMESTAMP=$(date +%Y%m%d-%H%M%S)
mv ./playwright-failures ./playwright-failures-$TIMESTAMP
echo "Previous report moved to: ./playwright-failures-$TIMESTAMP"

# Option 3: Skip download, use existing
# Just proceed to Step 1.2

---

If input is a GitHub URL, download the artifact first:

# For GitHub Actions run URL (e.g., https://github.com/saleor/saleor-dashboard/actions/runs/21513974962)
RUN_ID="[extracted-from-url]"
gh run download $RUN_ID -n merged-blob-reports -D ./playwright-failures/downloaded

# For PR URL (e.g., https://github.com/saleor/saleor-dashboard/pull/6292)
PR_NUM="[extracted-from-url]"

# Get LATEST run (always use most recent, not just failed)
RUN_ID=$(gh run list --branch $(gh pr view $PR_NUM --json headRefName -q .headRefName) --limit 1 --json databaseId -q '.[0].databaseId')

# If no runs found, ask user if they want to trigger tests
if [ -z "$RUN_ID" ]; then
    # Use AskUserQuestion: "No runs found. Add label 'run pw-e2e' to trigger tests?"
    # If yes: gh pr edit $PR_NUM --add-label "run pw-e2e"
fi

gh run download $RUN_ID -n merged-blob-reports -D ./playwright-failures/downloaded

Then run the prepare script:

# Use downloaded folder if from GitHub, or original $ARGUMENTS if local file/folder
INPUT_PATH="./playwright-failures/downloaded"  # or "$ARGUMENTS" for local
bash scripts/prepare-report.sh "$INPUT_PATH" ./playwright-failures

This script:

1. Detects if reports need merging (runs npx playwright merge-reports if needed)
2. Parses the JSON/JSONL report
3. Extracts failures with full error details and attachment paths
4. Provides semantic hints (categories, domains) to assist grouping
5. Creates summary files

Step 1.2: Read and Analyze Failures

Read ./playwright-failures/summary.md
Read ./playwright-failures/failures-full.json

Step 1.2.1: CRITICAL - Read Screenshots AND Error-Context Files

The error-context files are GOLD for debugging! They contain the accessibility tree (DOM snapshot) at the moment of failure.

For each failure, READ BOTH:

1. Screenshot (PNG) - Shows what the user sees at moment of failure
2. Error-context (Markdown) - Shows what's ACTUALLY in the DOM

⚠️ LIMITATION: Screenshots May Miss Transient UI

Screenshots only capture the FINAL state at failure. Transient UI elements like:

• Success/error notifications (auto-dismiss after a few seconds)
• Loading spinners
• Tooltips
• Dropdown menus

...may have already disappeared by the time the screenshot was taken!

When to Analyze Traces (for transient UI)

If the failure involves:

• expectSuccessBanner or notification assertions
• "element not found" but you expected it to appear briefly
• Timing-related failures

Use the Playwright trace to see step-by-step snapshots:

# Open trace viewer in browser (interactive)
npx playwright show-trace [TRACE_PATH]

# Or extract trace contents for analysis
unzip -l [TRACE_PATH]  # List contents
unzip [TRACE_PATH] -d ./trace-contents  # Extract

The trace ZIP contains:

• 0-trace.trace - All actions with timestamps
• 0-trace.network - Network requests and responses (NDJSON format)
• resources/ - Screenshots at EACH STEP + response bodies (referenced by SHA1)

Checking for API/GraphQL Errors in Traces

Error toasts often appear because a GraphQL mutation/query failed!

To check network requests:

# Extract trace
unzip [TRACE_PATH] -d ./.trace-extracted
cd ./.trace-extracted

# List all GraphQL requests with status codes
cat 0-trace.network | jq -r 'select(.snapshot.request.method == "POST") | "\(.snapshot.response.status) \(.snapshot.request.url)"'

# Check ALL GraphQL responses for errors (response bodies are in resources/)
for sha in $(cat 0-trace.network | jq -r 'select(.snapshot.request.method == "POST") | .snapshot.response.content._sha1' 2>/dev/null); do
  if [ -f "resources/$sha" ]; then
    result=$(cat "resources/$sha" | jq -r 'if .errors then "❌ ERRORS: " + (.errors[0].message // "unknown") else "✅ OK: " + (.data | keys | join(", ")) end' 2>/dev/null)
    echo "$result"
  fi
done

# Or use trace viewer (interactive, shows Network tab)
npx playwright show-trace [TRACE_PATH]

**Common GraphQL error patter

---

Content truncated.