rebuild-leaderboard

Rebuild Leaderboard from Scratch

Use when raw benchmark outputs are gone (e.g., cluster wipe) and you need to re-evaluate all models that appear on latamboard.surus.lat.

Cluster: ssh cluster.surus.ddns.net — 4× RTX 3090 (24 GB each)

How this works

The HuggingFace dataset (mauroibz/leaderboard-results) still has the old processed scores. You're not replacing them — you're re-running each model, getting fresh scores, and merging them back with merge_and_publish. Publish after every model, not at the end. If a run crashes, you keep everything already published.

Phase 1 — Prep (local or cluster)

1a. Confirm HF_TOKEN is set

grep HF_TOKEN .env || echo "MISSING — add HF_TOKEN=hf_... to .env"

1b. Check the dataset name in config

grep "results:" configs/config.yaml
# Should be:  results: "mauroibz/leaderboard-results"
# If it says  results: "LatamBoard/leaderboard-results"  → fix it first

1c. See what's currently on the leaderboard

python3 -c "
import json
with open('outputs/publish/summaries/all_model_summaries.json') as f:
    data = json.load(f)
print(f'{len(data)} models on HF:')
for k in sorted(data): print(' ', k)
" 2>/dev/null || echo "(no local summaries — that's fine, they're on HF)"

Phase 2 — Models with existing configs (run these first)

These models have a config in configs/models/ and can be run immediately. All use --tasks latam_board regardless of what the config file says, to ensure consistent coverage (spanish + portuguese + translation + structured_extraction).

The run loop

Run one model at a time — each takes a full GPU card and finishes in 30–90 minutes depending on model size.

# Template — repeat for each model below
benchy eval --config configs/models/<CONFIG>.yaml --tasks latam_board
# note the run_id printed at the start, then:
python -m src.leaderboard.merge_and_publish --run-id <RUN_ID>

Model list

With 4× 3090s you can run up to 4 models in parallel by pinning each to a different card:

# Parallel example — pin each run to a specific GPU
CUDA_VISIBLE_DEVICES=0 benchy eval --config configs/models/zephyr-7b-beta.yaml --tasks latam_board &
CUDA_VISIBLE_DEVICES=1 benchy eval --config configs/models/llama3.1.yaml --tasks latam_board &
CUDA_VISIBLE_DEVICES=2 benchy eval --config configs/models/ministral8b.yaml --tasks latam_board &
CUDA_VISIBLE_DEVICES=3 benchy eval --config configs/models/DeepSeek-R1-Distill-Qwen-7B.yaml --tasks latam_board &
wait
# Then publish all four:
python -m src.leaderboard.merge_and_publish --run-id <ID1>
python -m src.leaderboard.merge_and_publish --run-id <ID2>
python -m src.leaderboard.merge_and_publish --run-id <ID3>
python -m src.leaderboard.merge_and_publish --run-id <ID4>

Note: Two-card configs (vllm_two_card) need adjacent GPUs. Check the config's vllm.provider_config before parallelising — don't put two two-card models on the same pair.

Phase 3 — Models without configs (need new configs)

These models are on the leaderboard but have no matching config in configs/models/. Create a config for each using the configure-model skill, then run the same way as Phase 2.

To create a missing config:

# Minimal config template — save as configs/models/<name>.yaml
cat > configs/models/MyModel.yaml << 'EOF'
model:
  name: "org/ModelName"   # exact HuggingFace repo path

vllm:
  provider_config: "vllm_single_card"
  overrides: {}

tasks:
  - "latam_board"
EOF

Then validate before running:

benchy validate --config configs/models/MyModel.yaml

Phase 4 — Verify

After all models are published:

# Count models now in the dataset
python3 -c "
from huggingface_hub import hf_hub_download
import json
p = hf_hub_download('mauroibz/leaderboard-results', 'leaderboard_table.json', repo_type='dataset')
data = json.load(open(p))
print(f'{len(data)} models on HF:')
for row in data: print(' ', row.get('full_model_name', row.get('model_name')))
"

Open latamboard.surus.lat and hard-refresh — all models should appear with scores. The leaderboard fetches live from HF so no redeploy is needed.

Tracking progress

The fastest way to see where you are mid-rebuild:

# Which runs have completed?
ls outputs/benchmark_outputs/ | sort

# Which are already published to HF?
ls outputs/publish/summaries/*_summary.json 2>/dev/null | wc -l

If a run fails

Don't stop. Skip the failed model, keep going, come back to it:

# Check what went wrong
cat outputs/benchmark_outputs/<run_id>/<model>/run_outcome.json | python3 -m json.tool | grep -E '"status"|"reason"'

# Common fixes:
# - OOM: switch to two-card config or reduce batch size
# - API error: check HF_TOKEN, model access permissions
# - Task not found: model config may need --tasks override

Failed models won't overwrite good scores — merge_and_publish only adds/ updates, never deletes existing HF entries.