nightly-pr-triage

Nightly PR triage

A scheduled GitHub Action (.github/workflows/update-registry.yml) runs every night, scrapes hub.harborframework.com/datasets, regenerates src/inspect_harbor/_tasks.py and docs/registry-listing.yml, and opens a PR titled fix: update Harbor registry tasks on the update-harbor-tasks branch. When new datasets appeared upstream, the bot auto-stubs them in docs/overrides.yml with categories: []. Someone then needs to fill in real values before merge — scripts/validate_overrides.py runs in CI and blocks the merge on empty stubs.

Local vs. remote. This skill runs two ways. Locally, you invoke it in Claude Code. Remotely, the nightly workflow posts an @claude comment on the PR and the Meridian dev agent (.github/workflows/claude.yml) follows this skill on GitHub, pushes the filled overrides to the branch, and posts @review. The remote agent can't ask questions mid-run, so wherever this skill says to check with a human, the remote agent instead leaves the item as an empty stub and notes the uncertainty in a PR comment / for the reviewer rather than guessing — the leftover empty stub keeps CI red, which correctly blocks merge until a human resolves it. Steps 1–7 are the triage; the merge and step 8 (docs publish) are always human — the remote agent stops after pushing and posting @review.

Steps

1. Find and check out the PR

gh pr list --repo meridianlabs-ai/inspect_harbor --search "update Harbor registry tasks" --state open

Check out cleanly — the bot force-pushes to update-harbor-tasks so a stale local copy will fail to fast-forward. Always reset first:

git switch main
git pull --ff-only
git branch -D update-harbor-tasks 2>/dev/null || true
gh pr checkout <pr-number> --repo meridianlabs-ai/inspect_harbor

2. Identify newly-stubbed slugs

The PR description lists them under "Action required: fill in categories for new datasets". Or pull from the diff directly:

git diff main..HEAD docs/overrides.yml | grep -B1 "categories: \[\]"

Each newly-stubbed entry looks like:

org/name:
  categories: []

3. Triage each stub: keep, exclude, or dedupe

Not every auto-stubbed slug is a real benchmark worth listing. Before researching, decide for each stub whether it should be kept (→ research + fill, the common case), excluded (junk), or deduped (a copy of one we already list). Excluded/deduped slugs go in docs/exclude.yml instead of getting categories filled in.

The live scrape in step 6 (validate_overrides.py) often surfaces more new slugs than the PR description lists — datasets land on the hub after the nightly bot ran. Triage those the same way. Run validate early to see the full set.

Exclude experimental / personal junk. People push half-finished or personal tasks to the hub that aren't published benchmarks. Tells:

• Slug looks like a working file, not a release: task1_v3_1_..., ..._train_patched / ..._eval_patched, version/scratch suffixes (-v1, -rolling).
• desc reads like an internal dev note, e.g. "Patched verifier rebuild: reworked Modal capability-contract checks ... Tasks identical except tests/verify.py."
• Tiny task count (a handful of samples) with a generic name.
• The org is an individual, not a project/company. Check it:

  gh api users/<org> -q '.type + " | " + (.name // "") + " | repos=" + (.public_repos|tostring)'
  

  type=User (a person) with no matching benchmark repo is a strong exclude signal; type=Organization (or a User whose repo is the canonical benchmark) is a keep signal. A slug whose org doesn't resolve at all (404) and reads like a dev artifact is junk.

When in doubt about whether something is a real benchmark vs. junk, don't guess: leave it as an empty stub and flag the uncertainty — ask the user when running locally; note it in a PR comment / for @review when running remotely. The empty stub keeps validate_overrides.py red, which correctly blocks the merge until a human resolves it.

Dedupe copies published under multiple orgs. The same benchmark sometimes appears under several org slugs (verbatim re-uploads). Keep the copy from the most credible / original author — the benchmark's actual authors, or the org that owns the upstream GitHub repo — and exclude the rest. Confirm they're really the same before dropping one:

uv run python -c "
from inspect_harbor import <func_a>, <func_b>
a, b = <func_a>().dataset, <func_b>().dataset
print('counts:', len(a), len(b))
print('inputs identical:', sorted(s.input for s in a) == sorted(s.input for s in b))
"

Identical inputs (even if sample ids are renamed) ⇒ duplicate. Example: xiaoboai/pawbench is a verbatim copy of agentscope-ai/pawbench (same 150 inputs, renamed ids) — we keep agentscope-ai (the author org that owns the PawBench repo) and exclude xiaoboai/pawbench.

How to exclude. Add a glob to docs/exclude.yml with a one-line comment saying why, prefer an org-wide glob (ashantanu/*, vmax-modal/*) for a junk account and an exact slug (xiaoboai/pawbench) for a single dedup. Then drop any stub the bot already added to docs/overrides.yml (pop it in the helper from step 5, or it'll linger as an orphan-warning). After regenerating, the slug disappears from _tasks.py and the listing. See the header of docs/exclude.yml for the existing patterns.

4. Research each new dataset

For each slug you're keeping, gather:

Order of operations for research:

1. Check docs/registry-listing.yml for the slug — it carries Harbor's own desc field. That's often enough context for categorization, and the default desc is usually fine to publish as-is. Read it, though — it's surfaced on the registry listing and the AISI evals browser, so if it's a useless placeholder (or missing), plan to override it. See "Writing descriptions" below.
2. Look at sibling entries in docs/overrides.yml (same org/... prefix, or same benchmark family) for naming/category patterns. Example: when scale-ai/swe-atlas-rf got auto-stubbed, sibling scale-ai/swe-atlas-qna and scale-ai/swe-atlas-tw already had Coding + repo: https://github.com/scaleapi/SWE-Atlas + title: SWE-Atlas (QnA) / (Test Writing) — same pattern applied directly.
3. WebSearch (<benchmark name> github) when the brand or repo isn't obvious from the description. Skip this for self-evident names.
4. Use gNucleus AI / Harvey AI style company-and-benchmark phrasing when both matter for findability.
5. When the domain is ambiguous, inspect a task input directly. Load one sample and read its prompt — descriptions can mislead. Example: gnucleus-ai/cad-bench's description says "100 parametric FreeCAD tasks" which sounds like pure CAD/Professional. The actual task prompt is "Write a FreeCAD Python script to answer.py that reproduces the part described below" — so it's code-gen into a CAD domain → [Coding, Professional], not [Professional] alone. The one-liner:

   uv run python -c "from inspect_harbor import <func_name>; t = <func_name>(n_tasks=1); print(repr(t.dataset[0].input[:300]))"
   

Category picking:

The canonical vocabulary lives at the top of docs/overrides.yml. Mirror it in scripts/validate_overrides.py:CATEGORY_VOCAB and inspect_ai's docs/evals/sync.py:CATEGORY_VOCAB (we don't own this — inspect_ai does — so don't invent new categories without coordinating with them).

Common mappings:

• Code-generation / agent benchmarks → Coding
• Math, reasoning puzzles → Reasoning (use Mathematics only for explicit math content like AIME)
• Legal, finance, medicine → Law / Finance / Medicine (often paired with Professional)
• Vision/text-multimodal → Multimodal
• Safety/jailbreak → Safeguards
• Cybersecurity → Cybersecurity

Use a secondary category when the benchmark spans clear domains (e.g. MichaelY310/devopsgym is [Coding, Professional] because DevOps is both code and a professional domain). Don't pile on for breadth — 1–2 strong categories beat 4 weak ones.

Title styling:

• Match the project's own canonical capitalization (AIME, GAIA, USACO, BFCL, RExBench, SimpleQA).
• Use parens for splits: KUMO (easy), SWE-Lancer Diamond (Manager), Reasoning Gym (hard).
• Hyphens stay; spaces are for words: SWE-bench Verified, DevOps-Gym, Terminal-Bench v2.
• Greek/Unicode is fine if it's the project's own form: τ³-bench for sierra-research/tau3-bench.
• Skip the override when the leaf slug is already a clean display name (e.g. runebench, vmax-tasks).

Function-name dedup:

The generator derives the Python task function from the full slug (org/name → org_name), so a slug whose name repeats its org produces a stuttering identifier: android-bench/android-bench → `android

Content truncated.