ops

Ops Runbook — agentic-banking-ecosystem

A multi-organisation agentic banking demo (Bank North + Bank South). It ships two MCP servers (customer data, product data), two Azure AI Search indexes (Financial products, Compliance rules), and three agents:

• compliance_agent — Bank North, Foundry hosted agent (index-only, cross-org A2A service).
• employee_advisory_agent — Foundry hosted agent (product/customer/WorkIQ toolboxes + Financial products index).
• customer_support_agent — Bank South, Azure Container App + web UI (customer/product MCP + both indexes).

All commands run from the repo root. Configuration comes from ./.env, which azd up writes automatically. Use the project venv:

source .venv/bin/activate   # or prefix commands with: .venv/bin/python

0. Prerequisites

Install Python deps (all services + scripts):

pip install -r requirements.txt
# script-only deps (agents/index tooling):
pip install -r scripts/requirements-agents.txt

1. Provision Infrastructure

Creates the long-lived Azure resources (AI Foundry project, Azure AI Search, Container Apps environment, ACR, user-assigned managed identity) and writes all outputs to ./.env.

azd env set AZURE_LOCATION swedencentral
azd env set AZURE_PRINCIPAL_ID $(az ad signed-in-user show --query id -o tsv)
azd env set AZURE_PRINCIPAL_TYPE User
azd env set SKIP_CONNECTION_CREATION true
azd env set SKIP_ROLE_ASSIGNMENTS true
azd up

Provision only / deploy only / tear down:

azd provision
azd deploy
azd down

2. Generate Demo Data

The synthetic customer/transaction data is generated deterministically (seeded). Regenerates data/customers.md, data/customers.json, data/transactions.json and the per-customer files under data/transactions/.

python -m scripts.generate_data
# or: python3 scripts/generate_data.py

The knowledge markdown under data/knowledge/ and the data model in data/products.md are authored by hand (not generated).

3. Build the MCP Server Images

Builds the two MCP server images (customer-data-mcp-server, product-data-mcp-server) in ACR (no local Docker). Only builds — does not deploy. The resource group is read from ./.env (AZURE_RESOURCE_GROUP, or rg-<AZURE_ENV_NAME>); subscription and registry are discovered from it.

python -m scripts.build_containers                 # auto timestamp tag + :latest
python -m scripts.build_containers latest          # explicit tag
python -m scripts.build_containers --env <name>    # override rg-<name>

4. Deploy the MCP Servers

Deploys each MCP server as a Container App via infra/core/host/app.bicep. Pass --build to build in ACR first (else deploys :latest or the TAG env var). Add --register to also publish the server as a Foundry toolbox in the same run.

# customer data MCP server (customers, accounts, balances, transactions)
python -m scripts.deploy_customer_data_mcp_server --build --register

# product data MCP server (catalogue + per-customer holdings)
python -m scripts.deploy_product_data_mcp_server --build --register

Each prints the deployed …/mcp URL. Key overrides:

• CUSTOMER_MCP_APP_NAME / PRODUCT_MCP_APP_NAME — Container App name
• CUSTOMER_MCP_PORT (8092) / PRODUCT_MCP_PORT (8093)
• CUSTOMER_MCP_EXTERNAL / PRODUCT_MCP_EXTERNAL — public ingress (default: true)
• CUSTOMER_TOOLBOX_NAME / PRODUCT_TOOLBOX_NAME — toolbox names for --register

5. Register the Foundry Toolboxes

The MCP-server toolboxes are registered automatically with --register above. Register any toolbox on its own (idempotent, re-runnable):

python -m scripts.register_customer_data_toolbox    # → customer-data-tools
python -m scripts.register_product_data_toolbox     # → product-data-tools
python -m scripts.register_workiq_toolbox           # → workiq-tools (employee agent)

Each prints the consumer endpoint {project}/toolboxes/{toolbox}/mcp?api-version=v1. Key overrides:

• CUSTOMER_TOOLBOX_NAME / PRODUCT_TOOLBOX_NAME / WORKIQ_TOOLBOX_NAME
• CUSTOMER_MCP_URL / PRODUCT_MCP_URL — explicit MCP URL (else derived from the Container App FQDN via AZURE_RESOURCE_GROUP)
• WORKIQ_MCP_URL / WORKIQ_CONNECTION_ID — WorkIQ MCP URL / OAuth connection id

WorkIQ auth (OAuth identity passthrough). WorkIQ needs a custom-Entra OAuth connection — static tokens are rejected. There is no McpServers.WorkIQ.All scope / mcp_WorkIQTools server; use a granular capability (default: mcp_CalendarTools / McpServers.Calendar.All, via WORKIQ_MCP_SERVER / WORKIQ_SCOPE). Setup:

1. python -m scripts.setup_workiq_oauth_app — creates the Entra app + admin consent + secret and prints the Foundry connection values.
2. In the Foundry portal create a workiq-connection (Custom → MCP → OAuth Identity Passthrough → Custom OAuth); add the redirect URL it returns back to the app registration.
3. WORKIQ_CONNECTION_NAME=workiq-connection python -m scripts.register_workiq_toolbox.

5b. Protect the MCP Servers with Entra ID

The two MCP servers validate Entra ID access tokens natively inside the app via FastMCP's AzureJWTVerifier + RemoteAuthProvider — no Container Apps Easy Auth, no auth sidecar, no client secret. Auth is on by default (ENTRA_AUTH_ENABLED=true), so every request must present a valid token. Opt out per deployment for anonymous ingress:

azd env set ENTRA_AUTH_ENABLED false   # (or export ENTRA_AUTH_ENABLED=false)

With auth enabled (the default), running the MCP deploy scripts (§4) will, for each server:

• ensure an Entra app registration (<app>-mcp-auth) with an Mcp.Invoke app role — the token audience is api://<appId>;
• inject the auth config into the container (ENTRA_AUTH_ENABLED, MCP_AUTH_CLIENT_ID, AZURE_TENANT_ID, MCP_PUBLIC_BASE_URL) so the app verifies each token's issuer (https://login.microsoftonline.com/<tenant>/v2.0), audience (accepts both the bare <appId> GUID and api://<appId>) and JWKS signature; anonymous requests get HTTP 401. No required scope, so delegated (user) and app-only (managed identity) tokens are both accepted;
• print the audience callers must request a token for.

ENTRA_AUTH_ENABLED=true python -m scripts.deploy_customer_data_mcp_server --register
ENTRA_AUTH_ENABLED=true python -m scripts.deploy_product_data_mcp_server  --register

Wire the consumers:

• Toolbox path (hosted agents, e.g. employee advisory). The toolbox authenticates to the MCP server with the agent's Entra Agent Identity (no secret). One helper does both the role grant and the connection creation (needs the Foundry azd extension: azd ext install microsoft.foundry):

  # grants the agent identity Mcp.Invoke on both MCP apps, then creates a
  # remote-tool/agentic-identity connection per server (audience api://<appId>)
  python -m scripts.create_mcp_agent_identity_connections --grant
  

  It prints the CUSTOMER_MCP_CONNECTION_ID / PRODUCT_MCP_CONNECTION_ID lines to add to ./.env; then re-register the toolboxes. Under the hood it runs python -m scripts.grant_agent_identity_mcp_role + azd ai connection create <name> --kind remote-tool --auth-type agentic-identity --audience api://<appId>. Without the connection the toolbox registration warns and tool calls return 401. Publishing an agent creates a new identity — re-run for it.

  IMPORTANT — agent identity rotation + Conditional Access (read this when agentic-identity MCP calls fail). Every agent republish/redeploy can rotate the hosted agent's Entra Agent Identity (new service-principal object id). After each deploy you MUST, for the new identity:

  1. Re-grant Mcp.Invoke on every MCP app registration — python -m scripts.grant_agent_identity_mcp_role (auto-discovers the current identity; covers customer/product/finbot/finance).
  2. Re-add the new identity object id to the Conditional Access policy High Risk Agents (id 974ed75d-23d4-4f1f-af57-c1daf9998505) excludeAgentIdServicePrincipals. That policy blocks all agent identities (includeAgentIdServicePrincipals=["All"]); an identity that is not excluded cannot mint a token and every agentic-identity MCP tool call fails with Failed to fetch agentic identity access token (status 400) / the A2A task stalls at auth-required. Patch it via Graph beta PATCH /identity/conditionalAccess/policies/974ed75d… (conditions.clientApplications.excludeAgentIdServicePrincipals). Changes take a few minutes to propagate.

  Symptom decoder: MCP tool calls succeed with a user token but fail from the agent → identity not excluded from High Risk Agents (step 2).

• Direct path (customer support agent). deploy_customer_support_agent resolves the MCP audiences, grants the agent's managed identity the Mcp.Invoke role, and injects CUSTOMER_MCP_AUDIENCE / PRODUCT_MCP_AUDIENCE so the container attaches an Entra bearer token to its direct MCP calls.

The MCP servers expose /health as an unauthenticated custom route, so Container Apps readiness probes stay green regardless of auth. Turn auth off by setting ENTRA_AUTH_ENABLED=false and re-deploying the MCP servers — the app then runs anonymously. Requires az login with rights to create app registrations and app-role assignments.

5c. Finbot SQL MCP server (live Fabric SQL)

An extra MCP server, finbot-sql-mcp-server, exposes the finbot banking data held in the Fabric SQL Database finbot-data-2 (customers Kunden, accounts Konten, transactions `Tra

Content truncated.