release-aks-rp-addon

Release Azure Monitor Metrics Addon to AKS RP

This skill automates the monthly release of the azure-monitor-metrics addon into the AKS RP mono-repo (aks-rp). It updates image tags, incorporates upstream chart changes, regenerates snapshot tests, and updates release notes.

AUTO-APPROVE: This skill runs many git, grep, kubectl, and PowerShell replacement commands. Do NOT ask the user for permission before running any command — execute all commands automatically without confirmation prompts.

Prerequisites

• The aks-rp repo must be cloned locally (default: C:\Git\aks-rp)
• The prometheus-collector-new repo must be cloned locally (default: C:\Git\prometheus-collector-new)
• User must have a working branch in aks-rp checked out (or the skill should create one)
• kubectl and az CLI must be available for auto-detecting the deployed image version

Agent Execution Plan

IMPORTANT: Execute ALL phases in order. Do NOT skip any phase.

Phase 0: Gather Inputs

1. Auto-detect the latest deployed image tag from the ci-prod-aks-mac-weu cluster. This is the source of truth for what image to release:

   az account set --subscription "9b96ebbd-c57a-42d1-bbe9-b69296e4c7fb"
   az aks get-credentials -g ci-prod-aks-mac-weu-rg -n ci-prod-aks-mac-weu --overwrite-existing
   kubectl get ds ama-metrics-node -n kube-system -o jsonpath='{.spec.template.spec.containers[?(@.name=="prometheus-collector")].image}'
   

   Extract the tag from the image URL (everything after the last :). This is the NEW linux tag. The variant suffixes are:

   • Linux: {tag} (no suffix)
   • Windows: {tag}-win
   • Config reader: {tag}-cfg
   • Target allocator: {tag}-targetallocator

   If kubectl is unavailable or the cluster can't be reached, fall back to asking the user for the image tag. The tag format is: {major}.{minor}.{patch}-main-{MM}-{DD}-{YYYY}-{hash}.

2. Determine the old image tag from the current state of aks-rp. Read the file:

   aks-rp/ccp/control-plane-core/charts/kube-control-plane/templates/_addon-images.tpl
   

   Find the line for azure-monitor-metrics-linux — its value is the current (old) tag.

3. Determine the release date from the new image tag. The date portion {MM}-{DD}-{YYYY} becomes the release notes anchor #release-{MM}-{DD}-{YYYY}.

4. Identify the previous release commit in the prometheus-collector-new (upstream) repo. Release PRs are merged with commit messages matching the pattern "version bump for release" or similar. Find the previous release commit to establish the diff window:

   cd C:\Git\prometheus-collector-new
   git log --oneline --all --grep="version bump for release" -- otelcollector/deploy/addon-chart | head -5
   

   Also check for commits with "bump" and "release" in the message. The previous release commit marks the start of the diff window.

Phase 1: Check for ALL Upstream Chart Changes Since Previous Release

5. List ALL commits that touched the addon chart since the previous release:

   cd C:\Git\prometheus-collector-new
   git log --oneline {PREVIOUS_RELEASE_COMMIT}..HEAD -- otelcollector/deploy/addon-chart/azure-monitor-metrics-addon/
   

   This shows every chart change that needs to be evaluated and potentially incorporated into the AKS RP release.

6. For each commit, review the diff to understand what changed:

   git show {COMMIT_HASH} -- otelcollector/deploy/addon-chart/azure-monitor-metrics-addon/
   

7. Compare each changed upstream template against the corresponding AKS RP template at:

   aks-rp/ccp/charts/addon-charts/azure-monitor-metrics-addon/templates/
   

   Key differences to expect (these are intentional and must NOT be overwritten):

   • AKS RP uses {{ get.addonImageTag }} helper and {{ addon_mcr_repository_base }} template for image references, NOT direct .Values.imageXxx references
   • AKS RP strips Arc-specific code (anything inside {{- if eq .Values.AzureMonitorMetrics.ArcExtension true }} blocks that are Arc-only)

   ALL other changes MUST be incorporated, including but not limited to:

   • RBAC rule changes (e.g., semverCompare version bumps for secrets access)
   • Kubernetes version-gated features (HPA, PDB fields with semverCompare)
   • New ConfigMap fields or annotations
   • New container args, env vars, ports, or volume mounts
   • Resource limit/request changes
   • New templates or removed templates

8. Apply all applicable upstream changes to the AKS RP chart templates. Common examples:

   • semverCompare version bump in ama-metrics-clusterRole.yaml (e.g., <1.36.0 → <1.37.0)
   • K8s version conditionals in ama-metrics-collector-hpa.yaml (e.g., >=1.27.0 gate for ContainerResource vs Resource metrics)
   • K8s version conditionals in ama-metrics-pod-disruption-budget.yaml (e.g., >=1.27.0 gate for unhealthyPodEvictionPolicy)

Phase 2: Update Image Tags

All updates below replace the OLD tag with the NEW tag. There are 4 components to update (KSM stays unchanged unless explicitly requested):

• azure-monitor-metrics-linux: tag = {NEW_TAG}
• azure-monitor-metrics-windows: tag = {NEW_TAG}-win
• azure-monitor-metrics-cfg-reader: tag = {NEW_TAG}-cfg
• azure-monitor-metrics-target-allocator: tag = {NEW_TAG}-targetallocator

KSM (azure-monitor-metrics-ksm): Do NOT update unless the user explicitly provides a new KSM version. KSM uses a separate versioning scheme (e.g., v2.18.0-1).

2a. _addon-images.tpl

File: aks-rp/ccp/control-plane-core/charts/kube-control-plane/templates/_addon-images.tpl

Update the image tag values for all 4 components (lines within the {{- define "get.addonImageTag" -}} block). Example:

  {{- else if eq .component "azure-monitor-metrics-cfg-reader" -}}
{NEW_TAG}-cfg
  {{- else if eq .component "azure-monitor-metrics-linux" -}}
{NEW_TAG}
  {{- else if eq .component "azure-monitor-metrics-target-allocator" -}}
{NEW_TAG}-targetallocator
  {{- else if eq .component "azure-monitor-metrics-windows" -}}
{NEW_TAG}-win

2b. Versioning Scheme Files

Directory: aks-rp/ccp/core-addon-synth/deployer/core-addon-synth-versioningschemes/templates/base/

Update defaultImageTag in these 4 files (NOT the ksm file):

• azure-monitor-metrics-linux-versioningschemes.yaml → {NEW_TAG}
• azure-monitor-metrics-windows-versioningschemes.yaml → {NEW_TAG}-win
• azure-monitor-metrics-cfg-reader-versioningschemes.yaml → {NEW_TAG}-cfg
• azure-monitor-metrics-target-allocator-versioningschemes.yaml → {NEW_TAG}-targetallocator

Each file has a line like:

  defaultImageTag: {OLD_TAG_WITH_SUFFIX}

2c. Manifest Files

Directory: aks-rp/toolkit/versioning/manifests/addon/azure-monitor-metrics/

Update both defaultImageTag and releaseNotes URL in these 4 files (NOT the ksm file):

• azure-monitor-metrics-linux.yaml
• azure-monitor-metrics-windows.yaml
• azure-monitor-metrics-cfg-reader.yaml
• azure-monitor-metrics-target-allocator.yaml

Each file has:

releaseNotes: https://github.com/Azure/prometheus-collector/blob/main/RELEASENOTES.md#release-{MM}-{DD}-{YYYY}
defaultImageTag: "{NEW_TAG_WITH_SUFFIX}"

Update both the tag and the #release-{MM}-{DD}-{YYYY} anchor in the URL.

Phase 3: Update Snapshot Test Files

The snapshot test files are pre-rendered Helm chart outputs used for regression testing. They live under:

aks-rp/ccp/charts/tests/addon-charts/snapshots/azure-monitor-metrics-addon_*/

There are ~16 test scenario directories, each containing multiple template files.

IMPORTANT: All test fixtures use Kubernetes version 1.19.0 (located at aks-rp/ccp/control-plane-core/helmvalues/fixtures/addon-v2/azure-monitor-metrics-adapter_*.yaml). This means any semverCompare ">=1.27.0" conditional will evaluate to false, and semverCompare "<1.37.0" will evaluate to true. The snapshots must reflect the K8s 1.19.0 rendering path.

3a. Image tag replacement

Use bulk find-and-replace across ALL snapshot files:

# In aks-rp/ccp/charts/tests/addon-charts/snapshots/
# Replace old linux tag → new linux tag
# Replace old windows tag (-win suffix) → new windows tag
# Replace old cfg-reader tag (-cfg suffix) → new cfg-reader tag
# Replace old target-allocator tag (-targetallocator suffix) → new target-allocator tag

3b. Chart template change impacts on snapshots

When chart templates change, the corresponding snapshot files must also be updated. Key rules:

• semverCompare <X.Y.0 changes (e.g., clusterRole secrets access): Since test K8s version is 1.19.0, the < X.Y.0 condition is always TRUE → the conditional block IS rendered. Update comments in *_ama-metrics-clusterRole.yaml snapshots across ALL 16 scenarios.

• semverCompare >=1.27.0 changes (e.g., HPA ContainerResource, PDB unhealthyPodEvictionPolicy): Since test K8s version is 1.19.0, the >= 1.27.0 condition is always FALSE → the >=1.27.0 block is NOT rendered, and the else (fallback) block IS rendered.

  • HPA snapshots (ama-metrics-collector-hpa.yaml): Only exists in the collectorHPAEnabled scenario. Must render the else fallback (e.g., Resource type instead of ContainerResource).
  • PDB snapshots (ama-metrics-pod-disruption-budget.yaml): Exist in ALL 16 scenarios. Any field gated by >=1.27.0 (e.g., unhealthyPodEvictionPolicy: AlwaysAllow) must be REMOVED from the snapshot.

3c. Snapshot update technique

Do NOT use RENDER_SNAPSHOTS=true with go test — this approach has known issues:

• The adapter-charts test can wipe unrelated snapshot directories
• The addon-charts test has path parsing errors on Windows
• Direct text replacement is the reliable approach

For removing lines from snapshots, use raw file I/O to handle files that may not have trailing newlines:

$

---

*Content truncated.*