merge-train-infra

Merge-Train Infrastructure

This skill covers the automation internals of the merge-train system. For contributor-facing guidance (creating PRs, labels, handling failures), see the merge-trains skill.

Automation Lifecycle

The merge-train system is fully automated via GitHub Actions in .github/workflows/merge-train-*.yml:

1. PR Creation (merge-train-create-pr.yml): Triggered on push to merge-train/* branches. Creates a PR targeting next (or v5-next for -v5 trains such as merge-train/spartan-v5 and merge-train/fairies-v5) with the ci-no-squash label (plus private-port-next for any train that targets v5-next, and ci-full-no-test-cache for merge-train/spartan, merge-train/spartan-v5, and merge-train/ci). Skips merge commits and commits already in the base branch.

2. Body Updates (merge-train-update-pr-body.yml): Triggered on push to merge-train/** and backport-to-*-staging branches. Updates the PR body with meaningful commits (those containing PR references like (#1234)). The body wraps the commit list in BEGIN_COMMIT_OVERRIDE / END_COMMIT_OVERRIDE markers. Backport/port staging PRs also call update-pr-body.sh inline from scripts/backport_to_staging.sh to handle the first-push case (where the PR doesn't exist yet when the workflow fires).

3. Next Integration (merge-train-next-to-branches.yml): Triggered on push to next and v5-next. A push to next merges next into each next-based train; a push to v5-next merges v5-next into the -v5 trains (merge-train/spartan-v5, merge-train/fairies-v5). Both go through scripts/merge-train/merge-next.sh, which takes an optional second argument for the source branch (defaults to next). Uses continue-on-error: true so a conflict in one branch does not block others. Skips branches whose PR already has auto-merge enabled.

4. Auto-Merge (merge-train-auto-merge.yml): Runs hourly via cron (0 * * * *). Calls scripts/merge-train/auto-merge.sh for merge-train (4-hour inactivity) and backport-train (BRANCH_PATTERN=backport-to-, 8-hour) branches. Uses separate GitHub tokens: AZTEC_BOT_GITHUB_TOKEN for API calls and MERGE_TRAIN_GITHUB_TOKEN for approvals. Will not auto-merge if the last merge-queue CI run failed or was cancelled.

5. Recreation & Wakeup (merge-train-recreate.yml): Triggered when a PR is closed (merged). If the merged PR's head branch starts with merge-train/, recreates the branch from the base branch (usually next). Then runs scripts/merge-train/wakeup-prs.sh to add the ci-wakeup-pr-after-merge label to all open PRs targeting the branch that have passed CI and have automerge enabled. This triggers a CI re-run (typically a no-op via tree-hash cache) so those PRs can proceed through the merge queue. The label is immediately removed by a step in ci3.yml so it can be re-applied on subsequent merges.

6. Failure Notification (merge-queue-dequeue-notify.yml): Triggered when a PR is dequeued from the merge queue. If the PR's head branch starts with merge-train/ and the PR was NOT merged, sends a Slack notification via ci3/merge_train_failure_slack_notify. That script also kicks off a ClaudeBox session to investigate/fix the dequeued PR (ci3/slack_notify_with_claudebox_kickoff), passing --repo "$GITHUB_REPOSITORY" so the session runs in the mode matching the repo the train lives on. When the train is on a private mirror (…-private), claudebox.yml selects private mode; otherwise it stays public. Without that repo hint a private-train fix session lands in public mode and cannot read the PR or open the fix.

Label-Driven Ports (backport.yml)

backport.yml (triggered on pull_request_target labeled/closed) cherry-picks a merged PR onto an accumulating staging branch, then opens/updates one staging PR into a target branch, driven by scripts/backport_to_staging.sh:

• backport-to-<branch> (e.g. backport-to-v5-next): target is <branch> (derived from the label), staging branch backport-to-<branch>-staging. Direction next → release line.

There is no automated forward-port from a release line into next: the labs half of the tree lives in aztec-node, so mark-v5-port-candidates.yml only labels PRs against v5/v5-next with port-to-aztec-node for a person to port there.

On cherry-pick conflict the workflow comments on the PR, posts to #backports, and dispatches ClaudeBox (.claude/claudebox/backport.md) with the staging branch to resolve manually. Staging PRs are auto-merged by the 8-hour jobs in merge-train-auto-merge.yml.

CI Integration Details

CI Mode Selection (.github/ci3_labels_to_env.sh)

Merge-train branches influence CI mode:

• merge_group events or ci-merge-queue label → merge-queue mode
• If the merge-group event is for merge-train/spartan-v5 → upgraded to merge-queue-heavy mode (10 parallel grind runs instead of 4)
• Target branch merge-train/barretenberg → ci-barretenberg mode

CI Concurrency (.github/workflows/ci3.yml)

group: ci3-${{ (startsWith(github.event.pull_request.head.ref, 'merge-train/') && github.run_id) || ... }}

Merge-train PRs get full concurrency (each run has its own unique group via github.run_id), while non-merge-train PRs share a group by branch name with cancel-in-progress.

Instance Postfix (.github/ci3.sh)

if [[ "${PR_HEAD_REF:-}" == merge-train/* ]]; then
    export INSTANCE_POSTFIX=${PR_COMMITS:-}
fi

Merge-train PRs get a unique instance postfix (commit count) to allow parallel EC2 instances.

CI Modes in bootstrap.sh

• ci-barretenberg: Only builds and tests barretenberg (AVM disabled)
• ci-barretenberg-full: Full barretenberg CI including acir_tests
• merge-queue: 4x AMD64 full + 1x ARM64 fast in parallel
• merge-queue-heavy: 10x AMD64 full + 1x ARM64 fast in parallel (used for merge-train/spartan and merge-train/spartan-v5)

Test History Tracking (ci3/run_test_cmd)

if [[ "$is_merge_queue" -eq 1 || ("${TARGET_BRANCH:-}" =~ ^v[0-9]) || ("${TARGET_BRANCH:-}" == merge-train/*) ]]; then
    track_test_history=1
fi

Failure Notification (ci3/bootstrap_ec2)

When a CI run fails on an EC2 instance, it calls merge_train_failure_slack_notify to send failure notifications to the appropriate Slack channel based on the branch name.

Creating a New Merge Train

1. Create a branch from the desired base (next for most trains; a release line like v5-next for a release-specific train) with naming pattern merge-train/{team}. For a v5-release train use the -v5 suffix (merge-train/{team}-v5): merge-train-create-pr.yml routes any *-v5 branch to a v5-next base automatically and adds the private-port-next label.
2. Add the branch to the loop in .github/workflows/merge-train-next-to-branches.yml. If it tracks a base branch other than next, also add that base to the workflow's push trigger and pass it as the second argument to merge-next.sh (see the v5-next → -v5 trains wiring)
3. For a base other than next that the *-v5 convention does not already cover, set the PR base in .github/workflows/merge-train-create-pr.yml. Either way, add a stale-check job that passes BASE_BRANCH in .github/workflows/merge-train-stale-check.yml
4. Add the branch-to-Slack-channel mapping in ci3/merge_train_failure_slack_notify
5. Optionally add CI mode overrides in .github/ci3_labels_to_env.sh and bootstrap.sh
6. Push code to the branch -- automation handles PR creation from there

Key Files Reference

Workflows

Scripts

CI Configuration

Other Scripts

Content truncated.