failure-classify

failure-classify

Triage failing tests so a real regression does not get lost among tests that are expected to fail (because they verify that the system rejects bad input).

When to use

• After every test run that produced at least one failing test.
• Before triaging a CI failure manually.
• In the report-render skill's input pipeline.

Do not use this skill to decide whether a regression is "important enough to fix". That is a human judgement. The skill only sorts.

Inputs

Output

Classification taxonomy

Seven classes. Order matters: the first matching pattern wins. A test name that matches no pattern falls into ASSERTION_FAILURE by default.

Algorithm

1. Extract failing test names from the log. Swift Testing emits two ✘ lines per failing test (one for the issue, one for the summary). The skill deduplicates while keeping the first-seen order.
2. Classify each name against the patterns, first-match-wins.
3. Emit JSON: list, counts, grouped.

Worked example (Swift)

Log:

✔ Test foo() passed after 0.001 seconds.
✘ Test barRejectsZero() failed after 0.001 seconds with 1 issue.
✘ Test bazShouldBehave() failed after 0.001 seconds with 1 issue.

Output:

{
  "failures": ["barRejectsZero", "bazShouldBehave"],
  "failures_by_class": {
    "EXPECTED_FAILURE": 1,
    "ASSERTION_FAILURE": 1
  },
  "failures_grouped": {
    "EXPECTED_FAILURE": ["barRejectsZero"],
    "ASSERTION_FAILURE": ["bazShouldBehave"]
  }
}

The first test is an expected-failure pattern (Rejects). The second is a real regression candidate.

Anti-patterns

• Treating EXPECTED_FAILURE as a green light. A failure classified as EXPECTED_FAILURE still failed. The classification only means the failure is the test's purpose, not that the system under test behaves correctly. The test author must read the message.
• Adding patterns that match everything. Patterns are first-match wins. A pattern like .* would swallow every test into one bucket.
• Assuming a runner that is not in the language list still works. The v0.1 patterns cover Swift Testing only. For other runners, the user must extend the patterns and add a self-test.

Cross-references

• report-render — consumes the JSON output and embeds it in the Markdown report.
• drill — uses this skill to verify the loop reacts to a real failure.
• drift-check — the previous step in the loop.