tbench

Terminal-Bench Integration

This directory contains the mux agent adapter for Terminal-Bench 2.0, using Harbor as the evaluation harness.

Quick Start

When user asks to run a tbench, generally assume they mean in CI via workflow_dispatch.

# Run full benchmark suite
make benchmark-terminal

# Run specific tasks
make benchmark-terminal TB_TASK_NAMES="hello-world chess-best-move"

# Run with specific model and xhigh thinking
MUX_RUN_ARGS="--thinking xhigh" make benchmark-terminal TB_ARGS="--agent-kwarg model_name=anthropic/claude-opus-5"

# Run on Daytona cloud (high parallelism)
TB_ENV=daytona TB_CONCURRENCY=48 make benchmark-terminal

Daytona Cloud Sandboxes

For faster benchmarks, use Daytona cloud sandboxes instead of local Docker:

# Set API key (get from https://app.daytona.io)
export DAYTONA_API_KEY="your-api-key"

# Run with 48 concurrent cloud sandboxes (~6x faster than local)
make benchmark-terminal TB_ENV=daytona TB_CONCURRENCY=48

# Run specific tasks on Daytona
make benchmark-terminal TB_ENV=daytona TB_CONCURRENCY=48 TB_TASK_NAMES="chess-best-move stockfish-elo"

Account limits (Tier 3): Pool of 250 vCPU / 500GB RAM. Most tasks require 1 vCPU / 2GB RAM, with a few needing up to 4 vCPU / 8GB RAM. Harbor automatically requests the correct per-task resources.

Speed comparison:

Configuration

Environment Variables

• TB_DATASET: Dataset to use (default: [email protected])
• TB_CONCURRENCY: Number of concurrent tasks (default: 4)
• TB_TIMEOUT: Global timeout in seconds (default: 1800 = 30 minutes)
• TB_ENV: Environment to run in (local or daytona)
• TB_TASK_NAMES: Space-separated task names to run (default: all tasks)
• TB_ARGS: Additional arguments passed to harbor
• MUX_RUN_ARGS: CLI flags passed directly to mux run inside the container (e.g., --thinking high --use-1m --budget 5.00). This is the primary mechanism for all mux run flags — avoids per-flag plumbing.
• MUX_RUN_AS_GOAL: When set to 1, runs each task instruction as a strict mux run --goal objective while still piping the instruction to stdin. Use MUX_RUN_ARGS for goal limits such as --goal-turns and --goal-budget. Incomplete strict-goal exits are left scoreable so Harbor can verify the workspace.

Timeout Handling

The benchmark uses Harbor's global timeout applied to all tasks. The default is 30 minutes (1800 seconds), which provides sufficient time for most tasks while catching genuinely stuck agents. The mux runner does not wrap the command in GNU timeout; Harbor must classify task timeouts as AgentTimeoutError so the workflow can distinguish timeout/infra cases from mux process failures.

Design Rationale:

Based on analysis of Oct 30, 2025 nightly runs:

• Longest successful task: blind-maze-explorer-algorithm.hard at 20 minutes
• 95th percentile: ~15 minutes
• Mean duration: ~6 minutes

The 30-minute default provides comfortable headroom for complex tasks without excessive wait times for failed attempts.

Override timeout:

# Run with 60 minute timeout for very complex tasks
TB_TIMEOUT=3600 make benchmark-terminal

# Run with shorter 10 minute timeout for quick iteration
TB_TIMEOUT=600 make benchmark-terminal TB_SAMPLE_SIZE=5

Note: We prefer global timeout defaults over per-task configuration to avoid complexity and maintenance burden. If you find tasks consistently timing out, increase TB_TIMEOUT rather than adding per-task configuration.

Agent Configuration

The agent adapter accepts a few Harbor kwargs (passed via --agent-kwarg):

• model_name: Model to use (e.g., anthropic/claude-opus-5, openai/gpt-5.6-sol)
• experiments: Experiments to enable, comma-separated (e.g., programmatic-tool-calling)

All other mux run CLI flags (thinking level, mode, runtime, budget, etc.) are passed via MUX_RUN_ARGS — no per-flag plumbing needed.

CI dispatch (primary method):

# Run with model, thinking, and 1M context
gh workflow run terminal-bench.yml \
  -f model_name=anthropic/claude-opus-5 \
  -f mux_run_args="--thinking high --use-1m"

# Run GPT-5.6 Sol with budget cap and high thinking
gh workflow run terminal-bench.yml \
  -f model_name=openai/gpt-5.6-sol \
  -f mux_run_args="--thinking high --budget 5.00"

Strict goal-mode runs:

# Run a single task as a strict CLI Goal Run
MUX_RUN_AS_GOAL=1 \
MUX_RUN_ARGS="--thinking high --goal-turns 30 --goal-budget 10.00" \
make benchmark-terminal TB_TASK_NAMES="chess-best-move"

# CI dispatch
gh workflow run terminal-bench.yml \
  -f model_name=anthropic/claude-sonnet-4-5 \
  -f task_names=chess-best-move \
  -f mux_run_as_goal=true \
  -f mux_run_args="--thinking high --goal-turns 30 --goal-budget 10.00"

Local runs:

# Pass flags via MUX_RUN_ARGS env var
MUX_RUN_ARGS="--thinking high --use-1m" make benchmark-terminal

# Model and experiments via TB_ARGS
MUX_RUN_ARGS="--thinking high" make benchmark-terminal TB_ARGS="--agent-kwarg model_name=openai/gpt-5.6-sol --agent-kwarg experiments=programmatic-tool-calling"

Monitoring local benchmark output

Local Terminal-Bench runs can be long and log-driven. If you intentionally run one locally instead of dispatching CI, start it as a monitored background bash so Mux wakes on terminal benchmark failure/completion lines instead of requiring parent-side polling.

bash({
  script: 'make benchmark-terminal TB_TASK_NAMES="hello-world"',
  display_name: "Terminal-Bench Local",
  run_in_background: true,
  timeout_secs: 7200,
  monitor: {
    filter: "FAILED|ERROR|Traceback|AgentTimeoutError|results saved|Results saved|pass rate|score",
    cooldown_ms: 1000,
    max_events: 5,
  },
});

Use this only for line-oriented local process output. For GitHub workflow dispatch status, use a bounded background task/workflow monitor that polls gh/GitHub state.

Results

Results are saved to runs/YYYY-MM-DD__HH-MM-SS/:

• results.json: Aggregate results with pass/fail rates
• run_metadata.json: Run configuration and metadata
• <task-id>/: Per-task directories containing:
  • sessions/agent.log: Full agent execution log
  • sessions/agent.cast: Asciinema recording of agent session
  • sessions/tests.log: Test execution output
  • results.json: Per-trial results

CI/CD Integration

Querying Results from BigQuery

Mux Terminal-Bench results are uploaded to BigQuery after CI runs. Query via bq CLI after authenticating with gcloud auth login and setting project to mux-benchmarks.

Table: mux-benchmarks.benchmarks.tbench_results

Schema: run_id (STRING), task_id (STRING), model_name (STRING), thinking_level (STRING: off/low/medium/high), mode (STRING: plan/exec), dataset (STRING), experiments (STRING), passed (BOOL), score (FLOAT), n_input_tokens (INT), n_output_tokens (INT), github_run_id (INT), github_sha (STRING), ingested_at (TIMESTAMP).

See .github/workflows/terminal-bench.yml and .github/workflows/nightly-terminal-bench.yml for GitHub Actions integration.

Nightly workflow runs both Claude and GPT models on the full task suite, uploading results as artifacts.

Leaderboard Submission

To submit mux results to the Terminal-Bench 2.0 leaderboard:

Step 1: Prepare Submission

The leaderboard computes pass@k from multiple attempts per task. Provide multiple runs so each becomes its own job folder inside the submission.

# Download latest 5 successful nightly runs (recommended for submission)
python3 benchmarks/terminal_bench/prepare_leaderboard_submission.py --n-runs 5

# Use specific run IDs (each becomes a separate job folder)
python3 benchmarks/terminal_bench/prepare_leaderboard_submission.py --run-id 111 222 333 444 555

# Use multiple existing artifact directories
python3 benchmarks/terminal_bench/prepare_leaderboard_submission.py --artifacts-dir ./run1 ./run2

# Download latest single run (quick iteration)
python3 benchmarks/terminal_bench/prepare_leaderboard_submission.py

# Only prepare specific models
python3 benchmarks/terminal_bench/prepare_leaderboard_submission.py --n-runs 5 --models anthropic/claude-opus-5

This creates a properly structured submission folder at leaderboard_submission/ containing:

submissions/terminal-bench/2.0/Mux__<model>/
  metadata.yaml       # Agent and model info
  <job-folder-1>/     # Results from run 1
    config.json
    result.json
    <trial-1>/
      config.json
      result.json
      agent/
      verifier/
    ...
  <job-folder-2>/     # Results from run 2
    ...

Step 2: Submit via HuggingFace Python API

The hf upload CLI tends to timeout on large submissions due to LFS file handling. Use the Python API with an extended timeout instead:

# Install huggingface_hub (via uv or pip)
pip install huggingface_hub

# Authenticate (one-time setup)
hf auth login

import httpx
from huggingface_hub import HfApi
from huggingface_hub.utils import configure_http_backend

configure_http_backend(
    backend_factory=lambda: httpx.Client(timeout=httpx.Timeout(300.0, connect=60.0))
)

api = HfApi()
api.upload_folder(
    repo_id="alexgshaw/terminal-bench-2-leaderboard",
    folder_path="./leaderboard_submission/submissions",
    path_in_repo="submissions",
    repo_type="dataset",
    create_pr=True,
    commit_message="Add Mux + <Model> submission",
    commit_description="- Agent: Mux (Coder)\n- Model: <model>\n- <N> tasks × <K> attempts",
)

The PR will be automatically validated by the leaderboard bot. Once merged, results appear on the leaderboard.

Tips from past submissions:

• The prepare script already

Content truncated.