mcs-analysis-toolkit

mcs-analysis-toolkit

Overview

Six first-principles recipes for proving a claim about this repo instead of asserting it: derive it by hand from source, confirm with read-only tools, and (for review claims) build a discriminating experiment that refutes a wrong belief. Each recipe ends in a worked example reproduced 2026-07-02, with a command to re-verify it.

When to use, and when NOT to

Use when: sizing a suite's worst-case context load before shipping, auditing whether a description will actually trigger, verifying a hook truly blocks before depending on it, sanity-checking a multi-agent cost estimate, deciding whether a counter is trustworthy enough to cite, or reviewing a "fix" before merge.

Do NOT use for: routine tool runs once you already trust them, that is mcs-diagnostics-and-tooling (inventory counts, budget reports, eval runs, exit-code tables). What counts as sufficient acceptance evidence is mcs-validation-and-qa. How a hunch becomes an accepted result over time is mcs-research-methodology.

Recipe 1: Token / context-load accounting

Formula: tokens = ceil(words * 1.33) (TOKEN_RATIO, pipeline/hooks/_utils.py:13). Worst-case simultaneous load, per specialist: coordinator + specialist + its single largest reference file (only one specialist and one reference load at runtime). Hard ceiling: 5,500 tokens (pipeline/config/budgets.json: max_simultaneous_tokens), overridable per suite. Root CLAUDE.md:57's "5,000" is a stale outlier; enforcement uses 5,500.

Procedure:

• wc -w the coordinator, specialist, and largest references/ file; apply the formula to each, round up, sum.
• Check budgets.json.overrides for a key equal to the exact relative path the hook computes, not a guessed path.
• Confirm: python3 pipeline/hooks/check_context_load.py <coordinator SKILL.md> (hard hook).

Worked example (skills/soc-security/skills/threat-model-skill): 468 + 2828 + 1275 words → 623 + 3762 + 1696 = 6081 tokens, 581 over ceiling. A 6,100-token override exists in budgets.json but never matches the hook's computed key (nested-suite path mismatch). Full evidence: references/worked-examples.md §1.

Recipe 2: Trigger-reliability analysis

Formula (pipeline/specs/SKILL-TRIGGER-RELIABILITY-SPEC.md §1.2): activation directive + positive triggers + 2-5 quoted phrases + a negative boundary naming the alternative. Target 40-80 words (§1.5); under 20 "almost always" under-activates. The only hard floor is 10 words (check_frontmatter.py: MIN_DESCRIPTION_WORDS = 10), so 10-40 words passes silently.

Procedure:

• Count the description's words; score against the formula (activation verb? quoted phrases? negative boundary?).
• Sum always-loaded description cost library-wide: python3 pipeline/scripts/skill-audit.py --json (total_description_tokens, over_threshold).
• Design positive/negative eval pairs using the as-built shape from an existing department ({id, input, expected_skill, type, description}), not the spec's fuller markdown template (unimplemented).

Worked example: skill-audit.py --json over skills/: 123 skills, 5,966 description tokens, 0 over threshold. Under-target cluster: git-workflows (10-word description) and prompt-wizard (11 words) clear the floor but sit at a quarter of target, no quoted phrases or negative boundary; git-workflows is a coordinator (13 nested specialists) that per §1.6 should "cast a wide net" and does not. Full evidence: references/worked-examples.md §2.

Recipe 3: Hook-blocking verification experiment

Rule: a hook wired into settings.json is not proof it fires or blocks. Prove, in order: its event+exit-code contract (PreToolUse + exit 2 denies the tool, stderr feeds back as the reason; PostToolUse + any non-zero exit is non-blocking, whatever the script prints), which stream carries its message, then a live-fire test plus a control payload that should pass.

Procedure: Read the settings.json registration (event, matcher) and the script's stdin contract. echo '<crafted JSON>' | bash <hook.sh>, check exit code and stream. Repeat with a control payload, confirm exit 0. Trust the hook only after both pass.

Precondition (checked 2026-07-05): the live-fire result below needs a fresh (<ACCEPTANCE_GATE_STALE_HOURS, default 72; see mcs-change-control) contract with an unresolved | pending |/| failed | row. A stale or fully-resolved contract makes the payload exit 0 with no output, that is the staleness guard working, not the hook failing to block. Check the newest contract's mtime first (ls -lt .claude/council/sessions/*/acceptance-contract.md | head -1).

Worked example (hooks/acceptance-gate.sh): with a fresh (<72h) contract carrying a pending/failed row, a TaskUpdate → completed payload exits 2, message on stderr; a status: in_progress control exits 0. Same experiment would have caught the gate's 4-month decorative period: before 605112d (2026-06-22) the identical script was PostToolUse, printed to stdout, exited 1 (non-blocking), so completion always went through from creation (299a264, 2026-02-16). Full transcript: references/worked-examples.md §3.

Recipe 4: Cost/latency estimation for multi-agent runs

Model: the engine's table (commands/_council-engine.md §Cost Profiles & Model Routing) gives a 5-agent-baseline token range per mode × profile, scaled by selected_agents / 5: brainstorm 10-15K flat; quick 20-35K/30-50K/45-75K; standard 80-120K/120-180K/180-270K; deep = standard + 50-150K audit, for lean/balanced/max.

Procedure: Read mode + profile off the session; take the baseline range; multiply by agents_selected / 5. Label the result an estimate: do not convert to dollars from memory, the engine points to a README "Cost guide" for $/MTok that does not exist in README.md (verified 2026-07-02).

Worked example (session claude-config-model-optimization-20260702-0003): profile max, 7 agents, standard-mode artifacts. Baseline (5-agent, standard, max) = 180-270K tokens, scaled by 7/5 = 252K-378K tokens. Estimate, not a committed actual: no telemetry here captures real per-session totals as of 2026-07-02. Full derivation: references/worked-examples.md §4.

Recipe 5: Instrument-trust audit

Rule: before citing any counter, trace its write path: who writes it under what condition, prose or enforced code, does it land in a commit, can it be contaminated by something outside its scope.

Procedure: Grep for the field name in the code/prose that writes it; classify the step (prose vs. enforced code). git log -- <file>, diff HEAD vs. working tree, an uncommitted write is invisible to everyone else. Check for contamination (a machine-wide symlink shared across projects?). Trust the counter only after one full write-through-commit cycle by SHA.

Worked example (skills/council/registry.json): write step is prose only (commands/_council-engine.md:672-677), no code enforces it. Last commit before today: 729adcd (2026-04-24, all uses: 0); next: dc44611 (2026-07-02, 18 uses), a 69-day gap where the write never got committed. One session slug has no matching directory in this repo, because ~/.claude/skills symlinks here, so every workspace on this machine shares one file with no workspace tag. Cite it as "at least these sessions ran," never full usage history. Full evidence: references/worked-examples.md §5.

Recipe 6: Refute-first review

Rule: for any "fix," before approving it: state the exact wrong behavior it prevents as input → wrong output; build one discriminating experiment whose result differs pre-fix vs. post-fix; run it. Reading the diff and agreeing with its narrative is not verification.

Procedure: State the failure scenario in one sentence. Isolate the code path (inject a fake dependency/transport/stdin rather than the real network or tool). Run the same experiment against git show <fix_commit>^:<file> (pre-fix) and the current file (post-fix). Confirm the two disagree on a caller-visible field, not just internal structure.

Worked example (commit 898174d, mcp/openrouter/openrouter_client.py): claimed failure was that urlopen raises HTTPError on 4xx/5xx, misclassified as transport failure. Monkeypatching urlopen to raise HTTPError(code=429, ...) and reading consult(...)["error_kind"] gives "http" on the current file, "transport" on 898174d^, both reproduced directly: a caller-visible field changes, so the fix is genuinely load-bearing. PR #55 has zero GitHub reviews and comments: the review that mattered was this self-run experiment. Full script: references/worked-examples.md §6.

Gotchas

• WRONG: "the override is in budgets.json, covered." RIGHT: the key must equal the hook's computed relpath; skills/<suite>/skills/<specialist> almost never matches a flat skills/<specialist> key (Recipe 1).
• classify_file() treats any path containing a skills/ segment as "specialist," even repo-root skills/ itself, so nearly every top-level skills/*/SKILL.md gets the 1,500-word target regardless of intent. Only a sibling skills/ subdirectory makes it "coordinator" (git-workflows qualifies, prompt-wizard doesn't).
• Passing check_frontmatter.py proves only the 10-word floor, not that a description will trigger; trigger reliability has no hard hook.
• PostToolUse + non-zero exit is silently non-blocking; only PreToolUse + exit 2 denies a call. Confusing the two shipped a 4-month decorative hook (Recipe 3).
• A cost/token estimate from the engine's table is a range, not a telemetry-backed actual; nothing here records real per-session totals as of 2026-07-02.
• A counter present and non-zero may not be committed, or scoped to the project you're standing in; check both (Recipe 5).
• "Passed review" can mean zero human reviewers plus a local Claude review loop (gh pr view <n> --json reviews,comments); a merged-without-review fix still needs your own

Content truncated.