nic-ci-pipelines

NIC CI/CD Pipelines

Repository Split -- read this first

Workflow files are shared between two repositories, and every job is gated on github.repository. The same file behaves differently depending on which repo runs it.

Consequences:

• release-prep.yml / release-prep-lts.yml are internal-repo only. They are the only workflows that build release images. Dispatching them on the public repo is a no-op -- every job skips.
• release-publish.yml / release-publish-lts.yml are public-repo only. They never run docker build; oss-release.yml and plus-release.yml copy image manifests with skopeo from source_registry (default docker-mgmt-test.nginx.com) to the public targets.
• The public repo still builds images for PR/CI testing (ci.yml -> build-artifacts.yml) and again on merge (image-promotion.yml calls build-artifacts.yml with force: true before tagging edge/stable). Both push to the GCR dev registry -- they are test artifacts, not release artifacts.
• A publish failure is retryable on its own -- nothing needs rebuilding because the images already exist in the staging registry.

   internal repo                        public repo
   -------------                        -----------
   release-prep.yml                     release-publish.yml
     build-artifacts.yml                  oss-release.yml   (skopeo copy)
     push-prep-images  --> docker-mgmt-test.nginx.com --> GCR / Docker Hub / ECR Public / Quay / GHCR / NGINX Registry
     stage-helm-chart  --> oci://docker-mgmt-test...  --> publish-helm.yml (Helm repo + GHCR)
     binaries (SBOM, Cosign)                            certify-openshift-images (Pyxis)
     azure-upload      --> Azure blob                   operator (dispatch nginx-ingress-helm-operator/sync-chart.yml)
                                                        release-gate -> tag -> release-assets -> github-release

Workflow Architecture

The CI system uses GitHub Actions with extensive reusable workflow composition.

ci.yml (main CI orchestrator)                          [public repo]
  -> checks (format, lint, codegen, CRDs, chart version)
  -> verify-codegen (go mod tidy, make update-crds, make update-codegen, make telemetry-schema -- all must produce no diff)
  -> unit-tests, staticcheck, govulncheck
  -> build-artifacts.yml (reusable)   <- CI/test images only, pushed to GCR dev registry
       -> build-oss.yml (per-variant, matrix)
       -> build-plus.yml (per-variant, matrix)  <- also used for NAP variants
  -> package-tests, helm-tests
  -> setup-smoke.yml (reusable)
  -> smoke / e2e tests

image-promotion.yml (post-merge)                       [public repo]
  -> build-artifacts.yml (force: true)  <- rebuilds test images before promoting
  -> tags images edge/stable
  -> Trivy + DockerScout security scans
  -> publishes edge Helm charts to GHCR
  -> updates GitHub Release draft notes

release-prep.yml (dispatchable Stage 1: Creation)      [INTERNAL repo only]
  -> build-artifacts.yml (reusable)   <- the only release image build
  -> push-prep-images -> stages images in docker-mgmt-test.nginx.com
  -> stage-helm-chart -> stages Helm chart in oci://docker-mgmt-test.nginx.com/nginx-ic/helm
  -> binaries -> generates SBOM (Syft), signs (Cosign), creates tarballs
  -> azure-upload -> uploads signed tarballs to Azure blob storage

release-publish.yml (dispatchable Stage 2: Publish)    [PUBLIC repo only -- no builds]
  -> oss-release.yml (skopeo copy from source_registry: docker-mgmt-test.nginx.com)
  -> plus-release.yml (skopeo copy from source_registry: docker-mgmt-test.nginx.com)
  -> publish-helm.yml (publishes Helm chart to Helm repo & GHCR)
  -> certify-openshift-images (certifies UBI images on OpenShift / Pyxis)
  -> operator -> dispatches nginx-ingress-helm-operator/sync-chart.yml to raise the operator PR
  -> release-gate -> verifies all artifact publications succeed
  -> tag -> creates and pushes the vX.Y.Z git tag
  -> release-assets -> downloads tarballs from Azure and uploads to GitHub release draft
  -> github-release -> closes milestone and publishes the GitHub release draft

Two-Stage Release Architecture

Release pipelines are split into two independently dispatchable stages that run in different repositories:

1. Stage 1 (Prep / Creation) (release-prep.yml / release-prep-lts.yml) -- runs in nginx/kubernetes-ingress-internal:
   • Builds binaries and container images (the only place release images are built)
   • Stages container images and Helm charts in the internal test registry (docker-mgmt-test.nginx.com)
   • Generates Syft SBOMs, signs artifacts with Cosign, and uploads release tarballs to Azure blob storage
   • Does not create git tags, publish public images, or publish the GitHub release
2. Stage 2 (Publish) (release-publish.yml / release-publish-lts.yml) -- runs in nginx/kubernetes-ingress (public):
   • Copies prepped images from docker-mgmt-test.nginx.com to public registries (GCR, Docker Hub, ECR Public, Quay, GHCR, NGINX Registry) plus the marketplace registries (GCR Marketplace, ECR Marketplace, Azure Marketplace) for Plus
   • Publishes public Helm charts and certifies UBI images on OpenShift
   • Dispatches sync-chart.yml in nginx/nginx-ingress-helm-operator to raise the operator PR (requires a non-empty operator_version input, otherwise the job skips)
   • Verifies all prerequisites via release-gate
   • Creates and pushes the release git tag (vX.Y.Z or <lts_version>)
   • Downloads signed binaries from Azure blob storage and attaches them to the GitHub release draft
   • Closes the release milestone and publishes the GitHub release

Because the stages are decoupled and live in separate repos, a transient failure in publishing or external registry sync can be retried directly via release-publish.yml without rebuilding any images or binaries.

Key Workflows

Core CI & Testing

Release Workflows

Reusable Build Workflows (called via workflow_call)

Content truncated.