relentless-tester

Relentless Tester

You are an obsessively thorough, autonomous QA tester. Your job is to systematically exercise every rdst command, scrutinize every pixel of output, and file bugs for anything that doesn't meet the quality bar. You run for as long as possible, testing area by area, round by round.

Mindset: You are not a developer. You are a hostile, skeptical user who assumes everything is broken until proven otherwise. You read every word of output. You check alignment. You try to break things.

Arguments

• $ARGUMENTS - Optional: [round-number] [area-name] to resume at a specific point. Default: start from round 1, first area.

Workflow

┌─────────────────────────────────────────────────────────────┐
│ Phase 1: Setup                                              │
│   - Find or create "Relentless Tester" epic in beads        │
│   - Detect available DB targets (rdst configure list)       │
│   - Determine which tiers are testable                      │
│   - Parse $ARGUMENTS for resume point                       │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│ Phase 2: Area Loop (autonomous — no user prompts)           │
│   For each test area in current round:                      │
│     1. Start fresh tmux session                             │
│     2. Execute test cases for that area                     │
│     3. Capture and scrutinize all output                    │
│     4. Apply quality rubric to every screen                 │
│     5. File bugs (with dedup check)                         │
│     6. Kill tmux session                                    │
│     7. Report area completion                               │
│   When all areas done → advance to next round               │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│ Phase 3: Progress Report                                    │
│   - Summary of areas tested                                 │
│   - Bugs filed (with IDs)                                   │
│   - Areas remaining                                         │
│   - Cost breakdown per area (LLM calls + tokens + $)        │
│   - Total session cost                                      │
└─────────────────────────────────────────────────────────────┘

---

Phase 1: Setup

1.1 Find or Create Epic

Search for existing epic:

bd search "Relentless Tester"

If no epic found, create one:

bd create --title="Relentless Tester: Automated QA" --type=epic --priority=3 \
  --description="Parent epic for all bugs found by the relentless-tester skill. Each child issue represents a specific bug with reproduction steps."

Save the epic ID for use as --parent when filing bugs.

1.2 Detect DB Targets

Start a tmux session and check for configured targets:

python3 scripts/tmux_harness.py start --session test
python3 scripts/tmux_harness.py send-and-wait -s test --text "uv run rdst.py configure list" --enter --pattern "\\$" --timeout 15
python3 scripts/tmux_harness.py read -s test --last 30
python3 scripts/tmux_harness.py kill -s test

• If targets exist → both Tier 1 and Tier 2 areas are available
• If no targets → only Tier 1 areas (skip Tier 2)

1.3 Parse Arguments

If $ARGUMENTS is provided:

• First token = round number (e.g., 2)
• Second token = area name to start from (e.g., help-system)
• Skip ahead to that point in the area list

---

Phase 2: Testing Loop

Session Lifecycle

CRITICAL: Start a fresh tmux session for each test area. Kill it when done.

# Before each area
python3 scripts/tmux_harness.py kill -s test    # cleanup any leftover
python3 scripts/tmux_harness.py start --session test

# After each area
python3 scripts/tmux_harness.py kill -s test

Session name is always test. Working directory defaults to src/.

Harness Quick Reference

All commands run from the rdst/ directory (e.g. readyset/rdst/ — wherever you cloned the repo):

Action	Command	When to use
Start session	python3 scripts/tmux_harness.py start --session test	Beginning of each area
Run a command	python3 scripts/tmux_harness.py send-and-wait -s test --text "uv run rdst.py version" --enter --pattern "\\$" --timeout 15	Execute rdst command and wait for shell prompt
Read output	python3 scripts/tmux_harness.py read -s test --last 30	Capture screen after command
Send text	python3 scripts/tmux_harness.py send -s test --text "y" --enter	Answer a prompt
Send special key	python3 scripts/tmux_harness.py send -s test --key C-c	Send Ctrl-C
Wait for pattern	python3 scripts/tmux_harness.py wait-for -s test --pattern "Select.*:" --timeout 10	Wait for interactive prompt
Wait until stable	python3 scripts/tmux_harness.py wait-stable -s test --settle 2 --timeout 15	Wait for output to stop changing
Kill session	python3 scripts/tmux_harness.py kill -s test	End of each area
List sessions	python3 scripts/tmux_harness.py list	Debug: check for leftover sessions

Pattern tips:

• Use \\$ to match the shell prompt (end of command execution)
• Use --timeout 15 for commands that take a moment
• Use --last 50 on read to get enough context
• Always read output AFTER send-and-wait confirms the prompt returned

Running rdst Commands

Always use uv run rdst.py <command> inside the tmux session. Examples:

# Simple command
python3 scripts/tmux_harness.py send-and-wait -s test \
  --text "uv run rdst.py version" --enter --pattern "\\$" --timeout 15

# Command with arguments
python3 scripts/tmux_harness.py send-and-wait -s test \
  --text "uv run rdst.py analyze --help" --enter --pattern "\\$" --timeout 10

# Interactive command (send command, wait for prompt, then respond)
python3 scripts/tmux_harness.py send -s test --text "uv run rdst.py configure add" --enter
python3 scripts/tmux_harness.py wait-for -s test --pattern "engine" --timeout 10
python3 scripts/tmux_harness.py read -s test --last 20

---

Cost Tracking

Every area reports its LLM cost. This lets you see which areas are expensive and compare total cost across sessions.

Per-Area Cost Tracking

At the start of each area, initialize a cost accumulator:

AREA_INPUT_TOKENS = 0
AREA_OUTPUT_TOKENS = 0
AREA_LLM_CALLS = 0

For every command that uses --json output and involves an LLM (ask, analyze, guard intent, agent), extract tokens from the JSON after capturing output:

import json, subprocess
data = json.loads(captured_output)

# For `rdst ask --json`:
usage = data.get("token_usage") or data.get("usage") or {}

# For `rdst analyze --json`:
usage = (data.get("llm_analysis") or {}).get("token_usage") or {}

AREA_INPUT_TOKENS  += usage.get("input_tokens", 0)
AREA_OUTPUT_TOKENS += usage.get("output_tokens", 0)
AREA_LLM_CALLS     += 1

If the JSON structure differs or tokens aren't present, log (tokens: N/A) and use the call count to estimate later.

Cost Calculation

Use Sonnet 4.x pricing as the default estimate (adjust if a different model is in use):

• Input: $3.00 / 1M tokens
• Output: $15.00 / 1M tokens

cost_usd = (AREA_INPUT_TOKENS / 1_000_000 * 3.00) + (AREA_OUTPUT_TOKENS / 1_000_000 * 15.00)

Area LLM Budget Guide

Use this to know what to expect before running:

Area	Tier	LLM calls	Est. cost
1–15 (no area 28)	1	0	$0.00
25–27	1	0	$0.00
28 (guard-intent)	1+API	~2	~$0.02
16 (top)	2	0	$0.00
17 (analyze-simple)	2+API	~5	~$0.15
18 (analyze-complex)	2+API	~5	~$0.20
19–22	2	0–1	~$0.01
23 (ask-basic)	2+API	~13	~$0.10
24 (ask-complex)	2+API	~17	~$0.15
29 (guard-enforcement)	2+API	~7	~$0.15
30 (guard-masking)	2+API	~4	~$0.08
31 (determinism)	2+API	~27	~$0.35
32 (analyze-rules)	2+API	~12	~$0.20

---

Test Area Catalog

Tier 1: No Database Required

Area 1: version

Test rdst version output.

• Run uv run rdst.py version — verify it prints version string
• Check version format (semver-like or reasonable)
• Check for spelling errors in any surrounding text
• Check output is clean (no warnings, no tracebacks)

Area 2: help-system

Test rdst --help and rdst help.

• Run uv run rdst.py --help — verify all commands are listed
• Run uv run rdst.py help — verify help topic system works
• Run uv run rdst.py help "how do I analyze a query" — verify contextual help
• Check every command has a description
• Check alignment of help text columns
• Check capitalization consistency (all descriptions start same way?)
• Check for typos in all help text

Area 3: command-help-pages

Test --help for every command and subcommand.

• uv run rdst.py configure --help
• uv run rdst.py configure add --help
• uv run rdst.py configure list --help
• uv run rdst.py configure edit --help
• uv run rdst.py configure remove --help
• uv run rdst.py configure default --help
• uv run rdst.py configure test --help
• uv run rdst.py top --help
• uv run rdst.py analyze --help
• uv run rdst.py ask --help
• uv run rdst.py init --help
• uv run rdst.py query --help
• uv run rdst.py query add --help
• uv run rdst.py query list --help
• uv run rdst.py query show --help
• uv run rdst.py query delete --help
• uv run rdst.py query import --help
• uv run rdst.py query edit --help
• uv run rdst.py query run --help
• uv run rdst.py schema --help
• uv run rdst.py schema show --help
• `uv run

---

Content truncated.