Onboarding guide

Operational Scripts & Runbooks

This chapter covers every operational script and runbook in MMCA.ADC: the one-time Azure bootstrap, the database-per-service cutover story (how the legacy AtlDevCon monolith DB was split into four per-service databases, and why its scripts no longer exist), the disaster-recovery posture, the automated restore drill, the day-2 alert-triage runbook, the staged SQL managed-identity migration, the archive-and-drop of the legacy database, the Play Store asset pipeline, and the mobile store-submission runbook. For each artifact you will learn what it does, when an operator runs it, a step-by-step walkthrough with line cites, and why each gate or design choice exists. Architecture rubric categories are tagged inline; cross-links reach the IaC and CI/CD chapters and the ADRs that recorded the underlying decisions.

Architectural context. The database-per-service split (ADR-006) and the resilience/recovery posture (ADR-009) are the two decisions that make this chapter necessary. Read both ADRs before running any of these scripts against production.

Which repo. Every file cited here lives in MMCA.ADC. MMCA.Store ships same-named twins (infra/DISASTER-RECOVERY.md, infra/OPERATIONS.md, infra/SQL-MANAGED-IDENTITY.md, .github/workflows/dr-drill.yml, scripts/dr-restore-drill.ps1) with different contents. The quick tell: ADC's copies name the acc-rg resource group and the four ADC_* databases; Store's name ib_rg, MMCAStore and three Store_* databases (MMCA.Store/infra/DISASTER-RECOVERY.md:19, :26, :39-40). Reading the wrong one hands an operator the other application's recovery procedure.

What scripts/ actually contains today. Five files: azure-setup.sh, dr-restore-drill.ps1, play-store-capture.ps1, play-store-compose.ps1 and set-local-jwt-secrets.ps1. The last is a local-development helper rather than an operational runbook and is not walked here. The one-time cutover scripts and their workflow were deleted once the cutover completed (see the cutover section below), so do not go looking for them.

azure-setup.sh, One-time Azure bootstrap

File: MMCA.ADC/scripts/azure-setup.sh (175 lines)

What it is. A bash script that creates every Azure identity and OIDC credential the GitHub Actions deploy pipeline needs. It is idempotent: every step checks for existing state before creating anything (azure-setup.sh:12, "Idempotent: safe to re-run").

When to run. Once per environment: when standing up a fresh Azure deployment for the first time, or when rebuilding after a disaster recovery scenario where the identity objects were lost. Never run it as part of a regular deploy; deploy.yml consumes the outputs but never re-runs this script.

[Rubric §11, Security] assesses whether secrets are managed safely and whether OIDC / managed identities replace long-lived credentials. This script is the bootstrap for that posture: it creates a User-Assigned Managed Identity (UAMI) and federated GitHub OIDC credentials so the pipeline never holds a client secret.

[Rubric §17, DevOps & Deployment] assesses whether the provisioning and deployment pipeline is automated, repeatable, and documented. The bootstrap being a single idempotent script that also prints its own post-run checklist (azure-setup.sh:140-175) is the embodiment of that principle.

Walkthrough

Configuration block (azure-setup.sh:26-35). Hard-codes the target subscription (4513b073-3a04-4f5c-b272-bbcc329b2d49), tenant (c6369020-..., QiMata Technologies), resource group (acc-rg), location (eastus2), UAMI name (mmca-adc-github-deploy) and the GitHub coordinates (org ivanball, repo ADC, branch main, environment production). Edit these before running in a different environment. The inline comment at azure-setup.sh:32 is the one that saves an hour: the GitHub repo is named ADC, not MMCA.ADC, and an OIDC subject built from the wrong repo name authenticates nothing.

Why UAMI instead of an App Registration (azure-setup.sh:6-10). AAD App Registration creation is blocked in the QiMata tenant for non-admin users (Graph Authorization_RequestDenied). UAMIs are ARM resources, so RG-level Contributor plus role-assignment rights is sufficient to create them. The choice is forced by tenant policy, not preference.

Resource group (azure-setup.sh:43-47). az group create is a no-op when the group already exists. The comment (azure-setup.sh:40-42) notes that acc-rg is shared with unrelated production resources and that the create call is kept only for disaster recovery. That shared-RG fact is what makes mode=Complete forbidden for every Bicep deploy in this repo: Complete mode deletes resources absent from the template, which here would mean someone else's production.

UAMI creation and ID retrieval (azure-setup.sh:49-65). Creates the identity, then queries both clientId (azure-setup.sh:56-59) and principalId (azure-setup.sh:60-63) and echoes them. The clientId becomes the GitHub secret AZURE_CLIENT_ID; the principalId is what the role assignments below key on.

assign_role and the az-cli 2.84.x workaround (azure-setup.sh:69-100). Azure CLI 2.84.x returns a spurious MissingSubscription error on az role assignment create even when the write succeeds (azure-setup.sh:70-72). The function therefore hits the ARM REST API directly with az rest PUT (azure-setup.sh:89-98). It also checks for an existing assignment first (an az rest GET plus a JMESPath filter, azure-setup.sh:81-87), so a re-run is a no-op rather than a duplicate grant.

Role assignments (azure-setup.sh:102-105). Two roles are granted to the UAMI at RG scope:

• Contributor (GUID b24988ac-...), which deploys ARM/Bicep and manages Container Apps, SQL and the rest.
• AcrPush (GUID 8311e382-...), which pushes container images. The ACR admin user is disabled (DISASTER-RECOVERY.md:67-68), so the UAMI's AcrPush is the only image-push path that exists.

Note what is deliberately absent: Microsoft.Authorization/roleAssignments/write. That omission is a least-privilege decision with a visible downstream consequence, the avatar-storage grant has to be applied by hand (main.bicep:125-126, and section 6 of the mobile runbook below).

Federated credentials (azure-setup.sh:107-138). Two OIDC federated credentials are created on the UAMI through create_or_replace_fic, which skips creation when the credential already exists (azure-setup.sh:116-122):

• github-env-production, subject repo:ivanball/ADC:environment:production. The deploy job runs with environment: production, which is the subject GitHub mints.
• github-ref-main, subject repo:ivanball/ADC:ref:refs/heads/main. The fallback for workflow runs that do not bind to an environment.

Both are pinned to issuer https://token.actions.githubusercontent.com and audience api://AzureADTokenExchange (azure-setup.sh:129-131). The subject string is the entire security boundary here: it is why a fork, or any other repo, cannot mint a token Azure will accept.

Post-run checklist (azure-setup.sh:140-175). The script prints everything the operator must still do by hand: create the GitHub environment, add six required secrets (AZURE_CLIENT_ID, AZURE_TENANT_ID, AZURE_SUBSCRIPTION_ID, SQL_ADMIN_PASSWORD, JWT_RSA_PRIVATE_KEY_PEM, JWT_RSA_PUBLIC_KEY_PEM), one required variable (AZURE_RESOURCE_GROUP), then the optional secrets (OAUTH_GITHUB_CLIENT_ID, OAUTH_GITHUB_CLIENT_SECRET, ANTHROPIC_API_KEY, SMTP_PASSWORD) and optional variables (SMTP_HOST, SMTP_PORT, SMTP_FROM, SMTP_USERNAME). There is no HS256 fallback signing key in that list: RS256 is the only signing path, which is why both RSA PEMs are listed as required and deploy.yml:1329-1332 fails the run when either is missing.

One gap to know about before your first deploy. The checklist never mentions ALERT_EMAIL, but main.bicep:115-117 declares alertEmailAddress as a required parameter with @minLength(3) and no default (alerts that notify nobody are silent failures), and deploy.yml:1321-1324 refuses the run with an actionable error when vars.ALERT_EMAIL is unset. Follow the printed checklist exactly and the first deploy stops there. Set ALERT_EMAIL as well.

The database-per-service cutover, and why its scripts are gone

Before the surviving artifacts make sense, the story behind them does.

Before ADR-006. All four modules (Identity, Conference, Engagement, Notification) pointed at a single shared SQL database called AtlDevCon. That caused an outbox race: every service's OutboxProcessor polled the same dbo.OutboxMessages table and could claim another service's rows, producing duplicate dispatch (the defect documented in ADR-006, fixed 2026-06-07).

The ADR-006 decision. Adopt database-per-service. Each service owns ADC_Identity, ADC_Conference, ADC_Engagement or ADC_Notification, locally on the Aspire SQL container and in Azure as four Basic-tier databases on the same SQL server, generated from a single serviceDatabaseNames array (main.bicep:656-679, Basic 5 DTU with a 2 GB cap each). Cross-service references become scalar IDs (no cross-database foreign keys); CrossDataSourceDegradeConvention removes FK constraints at the EF level; consistency flows through the outbox and broker (ADR-003).

[Rubric §8, Data Architecture] assesses data modeling quality, isolation, and whether services own their own schema. The per-service database design addresses this directly: each service has its own schema, its own migrations project (Source/Hosting/MMCA.ADC.Migrations.SqlServer.{Identity,Conference,Engagement,Notification}), and its own dbo.OutboxMessages, so no service ever sees another's rows. The template states the ownership rule as an invariant rather than a convention: each service is the sole migrator of exactly one of these databases, and adding a service means adding its name to that array (main.bicep:651-654).

The three-commit rollout, and what remains of it. The Azure cutover was structured as three commits to prevent data loss: commit 1 provisioned the four ADC_* databases empty while the apps kept reading AtlDevCon; a one-time manual workflow then migrated and copied the data with row-count verification; commit 2 flipped the container-app connection strings; commit 3 downgraded AtlDevCon to the Basic archive SKU.

All of it landed, and the finished state is what the template now declares:

• Per-service connection strings are built once from a shared base and injected per app (main.bicep:160-174), reaching each container app as a Key Vault secret reference (main.bicep:1076 is Identity's DataSources__Identity__SQLServerConnectionString).
• The comment at main.bicep:160-163 records the invariant plainly: each service connects only to its own database, and AtlDevCon is no longer referenced by any app.
• Those four databases are now the whole application data estate, and the template says so in as many words (main.bicep:651-654).

AtlDevCon itself is gone (2026-09-02). The archive database that the cutover left behind was exported to the bacpac blob sql-archive/AtlDevCon-20260902.bacpac in storage account adcprodstpys4way4uzb3g, then dropped by hand, and its sqlDatabase resource was removed from the template. What sits at main.bicep:635-647 today is a comment block, not a resource: it records the drop, names the blob, and gives the restore path (az sql db import, about ten minutes). Reading that block is the fastest way to learn that the "NEVER delete" instruction older documents carry was deliberately superseded rather than forgotten. The archive-downgrade section below walks the full sequence.

The same comment block carries a warning worth internalizing before you touch anything in acc-rg: an unrelated SQL server named atldevcon in westus2 lives in this shared resource group, it predates MMCA, and it has never hosted an ADC database (main.bicep:645-647, repeated at POST-CUTOVER-atldevcon-downgrade.md:123-128). It is not the archive and must never be referenced, scaled or deleted as if it were. ADC's SQL server is adc-prod-sql-<token>.

The cutover scripts were deleted on purpose (2026-08-30, commit 6323a7b9). Four artifacts this chapter used to walk step by step no longer exist in the tree: .github/workflows/cutover-per-service-dbs.yml, scripts/copy-atldevcon-to-per-service-dbs.azure.ps1, scripts/migrate-atldevcon-to-per-service-dbs.ps1 and scripts/migrate-atldevcon-to-per-service-dbs.sql. They were single-use tooling whose own guard rails made a second run wrong rather than merely redundant: the copy skipped any target table that already had rows, so a run after the flip would silently do nothing while reporting success. Keeping a dangerous single-use script in scripts/ invites exactly that run. If a DR rebuild ever needs them, recover them from git history at that commit rather than reconstructing them from memory.

[Rubric §34, Architecture Governance] assesses whether the repository's contents reflect current policy rather than accumulated history. Deleting spent one-time tooling, and then retiring the archive database it produced once a bacpac served the same purpose, is that principle applied twice: first to scripts/, then to main.bicep. Note the ordering the template records (main.bicep:640-641): Incremental mode never deleted the database when the declaration left the template, so removing the resource and dropping the database were two separate, deliberate acts.

infra/DISASTER-RECOVERY.md, DR runbook

File: MMCA.ADC/infra/DISASTER-RECOVERY.md (221 lines; not the Store file of the same name)

What it is. The authoritative disaster-recovery runbook for the ADC production environment. Mandated by ADR-009: every consuming app must declare RTO/RPO per failure scenario, document the backup/restore mechanism, accept single-region risk in writing, and record restore drills in a drill-result table that cannot stay empty.

When to consult. On any data-loss event, corruption, failed deployment, or region outage. Also consult it when changing backup or retention settings, to see what the targets are. Day-2 alert triage (what to do when an SLO alert fires) is the companion infra/OPERATIONS.md, not this file (OPERATIONS.md:3-6).

Cross-link: see the IaC chapter for how infra/main.bicep provisions the LTR policies and alerts, and the CI/CD chapter for the deploy rollback mechanism and the freshness gates.

[Rubric §29, Resilience & Business Continuity] assesses whether the system has documented RTO/RPO targets and a drilled restore procedure. The DR file addresses both: the objectives table (DISASTER-RECOVERY.md:10-14) and a maintained drill ledger of six rows (DISASTER-RECOVERY.md:197-204), whose newest entry is a PITR restore of ADC_Conference on 2026-08-10 that completed in 2.1 min against the 2 h RTO and came back Online. The status block (DISASTER-RECOVERY.md:206-211) reads that ledger as a rotation rather than as a single success: every ADC_* database now carries a recovery proof, and the number it quotes is the slowest measured restore in the rotation (4.4 min), not the fastest. A second status paragraph is retained for the record (DISASTER-RECOVERY.md:213-221); it closes TD-10 on the strength of the original 2026-06-20 drill plus the Polly fault-injection tests in MMCA.Common and the Azure Monitor SLO workbook (main.bicep:772, the sloWorkbook resource embedding infra/workbooks/adc-slo-workbook.json). The ledger is filled by hand from the drill's job summary (DISASTER-RECOVERY.md:171-174), so its newest row can trail the workflow's run history; the deploy gate reads the run history, not the ledger.

[Rubric §11, Security] assesses credential hardening. The managed-identity section (DISASTER-RECOVERY.md:65-92) documents the out-of-band bootstrap for the adc-prod-apps-identity UAMI, the Key Vault (adckv<resourceToken>, RBAC-authorized) and both role grants (Secrets User for the apps, Secrets Officer for the deploy identity). The ACR admin user is disabled; runtime secrets reach the container apps as keyVaultUrl references, so no plaintext secrets exist in Container App environment variables. What lives in that vault is worth reading in full (DISASTER-RECOVERY.md:81-85): the SQL and Service Bus connection strings, the RSA/JWT keys, the SMTP, OAuth and Anthropic secrets, and the gateway synthetic-traffic key that lets the k6 load test bypass rate limiting. The bootstrap is out-of-band for the same least-privilege reason the avatar grant is: the deploy identity has Contributor but not role-assignment write, so main.bicep references the identity as existing rather than creating it (DISASTER-RECOVERY.md:70-72).

Three passages that have drifted from the template

Read these with the bicep open, because the runbook prose is behind the code in three specific spots. Nothing here makes the runbook unusable, but each one would mislead a first responder.

Alert resource type, count and cadence (DISASTER-RECOVERY.md:51-53). The prose says "three SLO metric alerts scoped to the App Insights component ... evaluated every 5 min over a 15-min window". Three things moved. The live rules are KQL log-search alerts: sloAlertSpecs (main.bicep:390-480) materialized as scheduledQueryRules named adc-prod-alert-<key>-v2 (main.bicep:482-486) and scoped to the Log Analytics workspace (main.bicep:501), not to the App Insights component. There are five of them, not three: the runbook's three request-path rules plus resilience-circuit-open (main.bicep:422-428) and ai-scoring-token-ceiling (main.bicep:468-476). And the default cadence is 15 minutes, not 5: evaluationFrequency and windowSize both fall back to PT15M (main.bicep:513, :515). The thresholds in the table (DISASTER-RECOVERY.md:55-59) are still correct for the three rules it lists.

The cadence change is a deliberate FinOps trade and the template shows its arithmetic (main.bicep:502-506): a scheduled-query rule is billed per evaluation, and the 5-minute tier costs about $1.47/month per rule against about $0.50 at 15 minutes. Because windowSize was already PT15M, each evaluation still looks at the same 15 minutes of data, so no threshold changed; what disappeared is the overlap between consecutive evaluations. The cadence fields are overridable per entry because one rule genuinely needs a different shape: the AI token ceiling evaluates a two-day window twice a day, both values pinned by ARM-side limits (main.bicep:454-476, :508-511). The one metric alert that remains is the Sev 1 gateway-availability rule (main.bicep:733-754), kept deliberately because availability has no status-code confound and never produced a false page (main.bicep:545-546).

[Rubric §13, Observability] assesses alerting and monitoring. The five SLO signals and their severities are: failed requests (sev 2, main.bicep:392-398), average server response time (sev 3, main.bicep:401-407), dependency failures (sev 2, main.bicep:410-416), a resilience circuit breaker opening (sev 2, main.bicep:422-428) and the AI scoring token ceiling (sev 3, main.bicep:468-474). The request-path queries carry tuning a metric alert could not express: 401 and 499 responses are excluded from the failure counts, SignalR hub connections are excluded from the average-duration rule because a hub reports its connection lifetime as request duration, and the two crawler etiquette paths (/robots.txt, /sitemap.xml) are excluded too, while a real 400, 404 or 500 burst still pages at the same threshold (main.bicep:373-389). The template names the exact incidents that forced each exclusion, which is the right level of evidence for a change that loosens an alert. Per-alert triage lives in infra/OPERATIONS.md below.

ALERT_EMAIL is no longer optional (DISASTER-RECOVERY.md:61-63). The runbook says the rules are created and visible in Azure Monitor even without the variable, "they just don't email". That state is no longer reachable: alertEmailAddress is a required parameter with @minLength(3) and no default (main.bicep:122-124), it feeds the action group (main.bicep:357), and deploy.yml:1253-1254 fails the deploy when vars.ALERT_EMAIL is unset. An alert that notifies nobody is now impossible in a deployed environment by construction, which is the stronger version of what the runbook was aiming at.

Deploy rollback description (DISASTER-RECOVERY.md:152-154). The runbook describes the post-deploy smoke gate as Gateway /health plus /.well-known/jwks.json plus the UI root. The live gate is broader in both dimensions: a revision-activation gate that requires the newest revision of every app to report Healthy, Running and 100% traffic weight, followed by six probes that reach every service through the Gateway (deploy.yml:1470-1481 for the reasoning, deploy.yml:1570-1580 for the probes). The activation gate exists because the HTTP probes alone cannot prove the new code is serving: a healthy Gateway keeps answering from the previous backend revision when the new one never goes ready, which is exactly how a readiness regression stayed hidden for four days (deploy.yml:1479). The rollback mechanism the runbook names, az containerapp revision copy, is still what runs (deploy.yml:1624).

Recovery objectives

The DR file targets three failure scenarios (DISASTER-RECOVERY.md:10-14):

These are deliberately modest: ADC is a regional, non-24x7 conference app. Sub-hour multi-region failover is explicitly not a goal (DISASTER-RECOVERY.md:16-17).

Accepted single-region risks

DISASTER-RECOVERY.md:19-36 lists three knowingly-accepted single points of failure. Note the topology it records (DISASTER-RECOVERY.md:21-23): one resource group (acc-rg), apps in the RG's region, and the SQL server in sqlLocation (westus2) because the QiMata Sponsorship subscription blocks new SQL servers in eastus2. That split is a bicep parameter, not an accident.

• One Azure SQL server, publicNetworkAccess: Enabled with the Azure-services firewall rule (main.bicep:826-835), mitigated by geo-redundant PITR and LTR. This is also where the runbook records the end of the archive: the AtlDevCon database was exported to a bacpac and dropped on 2026-09-02, and that bacpac was itself deleted permanently on 2026-10-03, so no pre-cutover rollback copy exists (DISASTER-RECOVERY.md:25-29).
• One Container Apps environment, all apps minReplicas: 1 outside conference mode. A zonal outage drops the app until Azure reschedules. The mitigation is no longer a judgment call made on the day: conference mode (conferenceMode, switched by the CONFERENCE_MODE_UNTIL repository variable on every deploy) keeps a second Gateway replica always on and holds the UI at its scaled maximum, so the two database-free front-door apps survive a single replica loss during the event; the first deploy after the window reverts it, and cost-guard.yml fails while an expired surge is still live (DISASTER-RECOVERY.md:30-35). The procedure is the "Conference day" section of OPERATIONS.md, walked below.
• One Service Bus namespace (Standard) and one ACR (DISASTER-RECOVERY.md:36).

Backup posture

Two tiers (DISASTER-RECOVERY.md:38-47), and nothing outside them any more:

• PITR: the Basic tier's 7 days of point-in-time restore on geo-redundant storage (the Azure default). Covers the "undo the last bad change" case with an RPO of minutes.
• LTR: long-term retention on all four live per-service databases, weekly P4W, monthly P12M and yearly P1Y at week 1, declared as the serviceDatabaseLtr resource (main.bicep:984).

The AtlDevCon bacpac that used to sit beside both tiers was deleted on 2026-10-03 (POST-CUTOVER-atldevcon-downgrade.md:103-110). One sentence in the runbook has not caught up: DISASTER-RECOVERY.md:45-47 still says the retired archive "lives in blob storage as a bacpac". Read the status banner of the post-cutover runbook as current.

Recovery procedures

Single database PITR restore (DISASTER-RECOVERY.md:124-129). Restore to a new name, validate, then rename or repoint via a redeploy. The worked example uses ADC_Conference.

LTR restore (DISASTER-RECOVERY.md:131-136). List available backups with az sql db ltr-backup list, then restore with az sql db ltr-backup restore.

Full region loss (DISASTER-RECOVERY.md:138-141). The deploy pipeline is region-parameterized (sqlLocation plus the RG location), so recovery is: create a new RG in a healthy region, geo-restore each ADC_* database there, then re-run deploy.yml pointed at the new RG. There is no longer a last-resort source for pre-cutover data: the runbook records that the bacpac was deleted on 2026-10-03 and that no pre-cutover restore path exists. The live ADC_* databases already hold that data, copied in at cutover (POST-CUTOVER-atldevcon-downgrade.md:105-110).

Storage account Shared Key (DISASTER-RECOVERY.md:143-150). The only consumer of the storage account key was the bacpac import (az sql db import --storage-key-type StorageAccessKey), so deleting the bacpac unblocked a hardening step: the apps reach blob storage as adc-prod-apps-identity over OAuth (FileStorage__ServiceUri, DataProtection__BlobStorageUri) and no workflow calls listKeys. The runbook records setting storageAllowSharedKeyAccess=false as a separate infra change still to be made, and warns that operators doing ad-hoc data-plane reads will need Storage Blob Data Reader once Shared Key is off.

[Rubric §11, Security] assesses credential hardening. An account key is a bearer credential to the whole account; recording which single procedure kept it alive is what turns "disable Shared Key" from a risky guess into a change with a known blast radius.

Restore drill

DISASTER-RECOVERY.md:161-183 defines the drill: PITR-restore a throwaway copy, confirm it comes back Online, record the measured restore time, then delete the copy. Only a copy is ever created, so the live databases are never touched. The file documents three ways to run it (DISASTER-RECOVERY.md:167-175) and names the scheduled one as the enforcing path:

• Scheduled (DISASTER-RECOVERY.md:167-170), the weekly cron, which rotates across the four live per-service databases by ISO week number so each earns a recovery proof roughly monthly. The retired AtlDevCon archive was never in the rotation and no longer exists as a database at all.
• One-click (DISASTER-RECOVERY.md:171-174), the dr-drill.yml workflow's manual workflow_dispatch, for a chosen database (the four live ADC_* databases, defaulting to ADC_Identity) and a chosen point in time. It prints the drill-result row in the job summary, ready to paste into the ledger.
• Local / CLI (DISASTER-RECOVERY.md:175), pwsh ./scripts/dr-restore-drill.ps1 -SourceDatabase ADC_Conference after az login.

All three wrap the same az sql db restore, verify, az sql db delete sequence (DISASTER-RECOVERY.md:177-183); there is no sqlcmd anywhere in the drill path.

The stated SLO is at least one successful drill per release train and after any backup or retention change, with the restore completing inside the 2 h RTO; a missed or failed drill is called a release-blocking regression for §29 (DISASTER-RECOVERY.md:185-187). The drill-result table (DISASTER-RECOVERY.md:197-204) is where that claim is cashed, and it carries six rows:

Read the shape of that table, not just the last row. The first entry is the pre-rotation drill in which one database stood in for all four; everything from 2026-07-20 onward is the weekly rotation, and those rows are the first recovery proofs ADC_Identity, ADC_Engagement and ADC_Notification ever had (DISASTER-RECOVERY.md:189-195). Each row also carries its dr-drill.yml run id, so every claim in the ledger is traceable to an Actions run instead of resting on the author's word. The spread is the other lesson: the same ADC_Identity database restored in 1.8 min in July and 4.4 min in August, which is why the status block quotes the slowest number against the 2 h RTO (DISASTER-RECOVERY.md:206-211). A ledger that keeps growing is what makes a claim like "restores take about two minutes" falsifiable; a single row cannot show variance at all. The same note (DISASTER-RECOVERY.md:193-195) records that rows naming AtlDevCon are kept as history now that the archive is gone, rather than being edited out.

The ledger stays honest because it is gated, not remembered: dr-freshness fails a deploy when the newest successful dr-drill.yml run is older than 8 days (deploy.yml:853, DISASTER-RECOVERY.md:209-211). The rotation itself is prose here but arithmetic in the workflow, which is the source of truth. The next section walks it.

dr-drill.yml and dr-restore-drill.ps1, the ADR-009 restore drill

Files: MMCA.ADC/.github/workflows/dr-drill.yml (119 lines), MMCA.ADC/scripts/dr-restore-drill.ps1 (101 lines)

What it is. The automation behind the drill requirement above: the workflow picks a target database and logs in to Azure via OIDC, the PowerShell script does the restore, the timing, the verification and the cleanup, and prints a drill-result row ready to paste into infra/DISASTER-RECOVERY.md (dr-restore-drill.ps1:3-6).

When it runs. Both on a schedule and on demand, and the scheduled half is the one that carries the §29 weight:

• Weekly cron, Mondays 06:00 UTC (dr-drill.yml:30-32). This is the enforcing path: a throwaway-copy restore is cheap and is deleted immediately after, so recovery is proven continuously rather than whenever somebody remembers (dr-drill.yml:7-8).
• workflow_dispatch (dr-drill.yml:14-29) for an ad-hoc drill against a specific database and restore point. The dispatch default is ADC_Identity with a 10-minutes-ago restore point.

The job holds id-token: write plus contents: read and nothing else (dr-drill.yml:34-36), which is the least privilege an OIDC login needs, and it logs in with the same three AZURE_* secrets azure-setup.sh printed (dr-drill.yml:60-65). Both third-party actions it uses are pinned to a full commit SHA with the human-readable version parked in a trailing comment: actions/checkout@3d3c42e5... (dr-drill.yml:58) and azure/login@a641126d... (dr-drill.yml:61).

[Rubric §32, Dependency & Supply-Chain] assesses whether third-party dependencies are pinned, scanned, and cannot change underfoot. A Git tag is mutable, so @v3 in a job that mints an Azure OIDC token means a repointed tag could run new code with production credentials; the SHA pin removes that path, and the # v7.0.1 / # v3.1.0 comments keep the pin readable and updatable.

Why the drill shares the deploy's concurrency group. The workflow joins the prod-azure group with cancel-in-progress: false (dr-drill.yml:42-44). Its comment gives the reason (dr-drill.yml:38-41): the drill creates and deletes a database on the production SQL server, and deploy.yml declares that every workflow mutating production Azure state joins that same group (deploy.yml:39-45), so a drill and a deploy (which migrates those very databases) never run at the same time. Never cancelling an in-flight drill matches the deploy side, for the reason the cleanup sweep below exists: a drill killed mid-restore never reaches the script's own cleanup. The cost is queueing. A push to main that lands during a drill waits behind it, for at most the drill's 60-minute timeout. And because GitHub Actions keeps only one pending run per concurrency group, a scheduled drill still waiting behind a deploy can be displaced by a newer pending deploy; the 8-day dr-freshness window, wider than the weekly cadence, is what absorbs one lost run, and a manual dispatch restores the proof.

[Rubric §17, DevOps & Deployment] assesses whether deployments are safe and repeatable. Serializing every production-mutating workflow on one group turns "do not drill during a migration" from an operator habit into a property of the pipeline.

Scheduled runs rotate across the four live per-service databases by ISO week number modulo 4 (dr-drill.yml:67-86, the arithmetic at dr-drill.yml:79-82), so each live database gets a recovery proof roughly monthly. Which branch runs is decided purely by whether the dispatch input is present (dr-drill.yml:73), which is what lets one job serve both triggers. The chosen database is echoed into the step summary before the drill starts (dr-drill.yml:86), so a reader of a failed run knows immediately which database was under test. Note how the input is read: the step declares DISPATCH_DATABASE: ${{ inputs.source_database }} as an environment variable (dr-drill.yml:69-70) and the shell tests "${DISPATCH_DATABASE:-}" (dr-drill.yml:73) instead of expanding the ${{ }} expression inline, and the whole script runs under set -euo pipefail (dr-drill.yml:72) so an unset variable or a failed az call stops the step rather than silently rotating to the wrong database.

How the inputs reach the PowerShell script. The drill step sets three variables in its own env block, DRILL_RESOURCE_GROUP from vars.AZURE_RESOURCE_GROUP, DRILL_SOURCE_DATABASE from the rotation step's output, and DRILL_RESTORE_POINT_MINUTES_AGO from the dispatch input with a literal '10' fallback for scheduled runs (dr-drill.yml:92-95), then passes those variables to the script (dr-drill.yml:96-101). The workflow's own comment states the reason (dr-drill.yml:90-91): values reach the script through the environment rather than being interpolated into the command line, so a dispatch input cannot inject extra PowerShell arguments. This matters more here than in most jobs, because the step runs with an Azure session that holds Contributor on the production resource group.

[Rubric §11, Security] assesses credential handling and injection resistance. Workflow expression interpolation is textual substitution into the shell command before the shell ever sees it, so a hostile input value becomes code; routing every input through env and quoting it turns the same value back into data. The type: choice input (dr-drill.yml:19-25) already constrains the database name to four options, so this is defence in depth rather than the only control.

Why the job gets 60 minutes. The job carries a 60-minute timeout-minutes (dr-drill.yml:56), and the comment above it records why it is not 30 (dr-drill.yml:49-55): the 2026-09-14 scheduled run (run 34838047421) was killed at the old 30-minute limit and recorded no freshness proof. A measured PITR restore of an ADC_* database takes minutes, but Azure SQL queues the restore on the platform side and its duration varies from run to run, so 60 leaves headroom for a slow one (MMCA.Store raised its own drill job to 60 after the same failure). The limit matters beyond the drill itself: the dr-freshness deploy gate consumes this workflow's last success, so a drill that times out does not just lose one measurement, it walks the next production deploy toward a hard block (the gate is walked below).

The sweep that runs even when the drill does not finish. A job cancelled at its timeout-minutes is killed mid-step, so the script's own finally cleanup never runs. The workflow comment records the incident that exposed this (dr-drill.yml:103-107): the same 2026-09-14 scheduled run was cancelled mid-restore and left ADC_Engagement-drill behind, and the weekly rotation meant the next run targeted a different database, so the script's name-specific stale-copy check would not have removed it either; the copy sat at Basic-tier cost for five days. The last step, Remove leftover drill copies, therefore runs under if: always() (dr-drill.yml:108-109), which fires on success, failure and cancellation alike. It resolves the server by the same adc-prod-sql- name prefix the script uses, exits 0 with a workflow warning when no server matches (dr-drill.yml:114-115), and deletes every database whose name ends in -drill (dr-drill.yml:116-119), not just the copy this run created. The suffix filter is what keeps the sweep safe on a production server: the live ADC_* databases never end in -drill, so the only names it can match are throwaway copies. One consequence for a local run: a copy kept with the script's -KeepCopy switch for a manual row-count check (dr-restore-drill.ps1:83-88 honors it) is deleted by the next workflow run, so finish that check before Monday 06:00 UTC. It reads the resource group through env like the drill step (dr-drill.yml:110-111) and runs under set -euo pipefail (dr-drill.yml:113).

AtlDevCon is gone from both ends of this pair. The dispatch choice input offers only the four live databases (dr-drill.yml:21-25), and the script's own -SourceDatabase default moved from AtlDevCon to ADC_Identity (dr-restore-drill.ps1:26). Both files record why: the archive was exported to a bacpac and dropped on 2026-09-02, so it can no longer be PITR-restored (dr-restore-drill.ps1:10-12), and the workflow header adds that the bacpac itself was deleted on 2026-10-03 (dr-drill.yml:10-12), so there is no recovery path to drill at all. Leaving it selectable would have offered an operator a drill that fails for a reason unrelated to recoverability, and that failure would then have blocked the next production deploy through dr-freshness. The older argument still holds too: proving a retired archive restores proves nothing about the databases that take writes.

[Rubric §29, Resilience & Business Continuity] is what this workflow serves: it converts the DR runbook from a document into a measurement.

Walkthrough of the script

Parameters (dr-restore-drill.ps1:23-30). -ResourceGroup (default acc-rg), -SourceDatabase (default ADC_Identity), -RestorePointMinutesAgo (default 10), -KeepCopy, and -SummaryPath (the workflow passes $env:GITHUB_STEP_SUMMARY, dr-drill.yml:88-101). The workflow never leans on a default: it passes an explicit -SourceDatabase from the rotation step and an explicit -ResourceGroup from vars.AZURE_RESOURCE_GROUP (dr-drill.yml:93-94, :98-99), so the script's own defaults only matter to a local CLI run.

Server discovery (dr-restore-drill.ps1:40-44). Queries by name prefix (adc-prod-sql-*) rather than hard-coding the resource-token suffix, and throws a clear "Did you run 'az login'?" error when nothing matches. Resolving from live Azure state is what lets the same script work against a rebuilt environment whose token differs.

Stale-copy cleanup (dr-restore-drill.ps1:52-57). Deletes any <SourceDatabase>-drill left by a previous run, so the restore name is free. Without it, one crashed run would fail every subsequent drill on a name collision, and the freshness gate would then start blocking deploys for a reason that has nothing to do with recoverability. The check only covers the database being drilled this run, which under the weekly rotation is usually a different one, so the workflow's always() sweep (dr-drill.yml:108-119) is what catches a copy left by an earlier run against another database.

Restore and timing (dr-restore-drill.ps1:59-66). A stopwatch brackets az sql db restore, and $LASTEXITCODE is checked explicitly (dr-restore-drill.ps1:65) because a failing az call does not by itself throw in PowerShell. That elapsed time IS the measured RTO reported in the drill row; it is not an estimate.

Verification (dr-restore-drill.ps1:68-76). ARM-level: az sql db show --query status must read Online, otherwise the drill is a FAIL. A deeper row-count check is deliberately out of scope and needs -KeepCopy plus a manual pass (dr-restore-drill.ps1:18-21).

Cleanup (dr-restore-drill.ps1:83-88). The delete sits in a finally block, so the throwaway copy is removed even when the restore or the verification threw. A finally block cannot run in a process that GitHub Actions kills at timeout-minutes, though, which is why the workflow backs it with the always() sweep described above (dr-drill.yml:103-119): between them, a weekly drill does not accrete paid databases.

Result row and exit code (dr-restore-drill.ps1:90-101). Prints the markdown table row (date, source, PASS/FAIL, minutes, note) to both the console and the job summary through the Write-Line helper (dr-restore-drill.ps1:35-38), then exits 1 on anything other than PASS (dr-restore-drill.ps1:101), so a failed drill is a failed workflow run and therefore a stale recovery proof.

What it does not do, and the gate that makes it matter

The drill never touches a live database and it does not gate production directly. The link to production is the age of its newest successful run, checked by the dr-freshness step of deploy.yml's single freshness job (deploy.yml:826-856). That job holds the four recency gates as four steps whose name and id are the gate names, and every step after the first runs unless the job was cancelled, so one stale proof never hides the state of the others (deploy.yml:826-830). The dr-freshness step calls the shared MMCA.Common composite action ivanball/MMCA.Common/.github/actions/freshness-gate@main (deploy.yml:849) with workflow: dr-drill.yml and window-days: '8' (deploy.yml:852-853). It names no required-jobs, so the action runs in its success mode: the newest run that completed with success, failing the deploy when that run is older than the window (MMCA.Common/.github/actions/freshness-gate/action.yml:16-26). The comment at deploy.yml:846-848 records why the gate can trust that run at face value: dr-drill.yml has no skip-if-unchanged guard, so every successful run really performed a PITR restore. The job itself asks for only actions: read plus contents: read (deploy.yml:836-838) and runs on a 5-minute timeout (deploy.yml:835), because it is a handful of API reads and no restore cost per deploy.

Why a shared action rather than inline bash: the DR, load, cross-service and cross-browser steps all call the same action (deploy.yml:849, :868, :891, :931), and so does MMCA.Store's deploy (action.yml:8-9), so the staleness arithmetic and the break-glass rules live in one place and a fix lands for every gate at once instead of in several drifting copies. The reference is the MMCA.Common main branch rather than a SHA pin like the drill's third-party actions: it is first-party code, and that branch only moves through a PR.

freshness sits in deploy.needs (deploy.yml:1159), and the deploy's condition requires it to have concluded success (deploy.yml:1193), which means all four steps passed. Break-glass exists but is deliberately expensive to use: the step forwards the skip_freshness_gates and skip_justification dispatch inputs (deploy.yml:13, :17) to the action (deploy.yml:855-856), which skips only with a non-empty justification and writes that justification loudly to the step summary and as a workflow warning (action.yml:6-8, :99).

The operational consequence for an on-call reader: a red or skipped weekly drill blocks the next production deploy. If a deploy fails on dr-freshness, the fix is to re-run dr-drill.yml (and fix it first if it genuinely failed), not to bypass it. The gate's full mechanics, the break-glass input and the sibling gates are documented in the CI/CD chapter; the decision is ADR-064.

infra/OPERATIONS.md, day-2 alert triage runbook

File: MMCA.ADC/infra/OPERATIONS.md (431 lines)

What it is. The alert-to-action companion to the provisioned observability: what to do when each alert fires, how to read the SLO workbook, how to run conference day, the standard recovery moves, and why production has no Aspire dashboard. It explicitly defers restore procedure, RTO/RPO and accepted SPOFs to DISASTER-RECOVERY.md and covers day-2 triage only (OPERATIONS.md:3-6).

When to consult. When an alert email arrives from the adc-prod-alerts-* action group (main.bicep:357), whose only receiver is the address in the ALERT_EMAIL repository variable (OPERATIONS.md:11-12), and about a week before a conference (OPERATIONS.md:320-361).

[Rubric §13, Observability & Operability] assesses whether alerts lead anywhere. A threshold with no runbook is a page nobody knows how to answer, which is exactly what this file, and the build gate below, exist to prevent.

The pairing rule, enforced at build time

ObservabilityConventionTestsBase (in the MMCA.Common.Testing.Architecture package) pairs the alerts declared in the consumer's bicep against the runbook sections in its OPERATIONS.md. ADC subclasses it in MMCA.ADC.Architecture.Tests and wires it by embedding both files as manifest resources named infra.main.bicep and infra.OPERATIONS.md (MMCA.ADC.Architecture.Tests.csproj:17-22). The base resolves those resources from the derived type's assembly (ObservabilityConventionTestsBase.cs:159), which is the detail that lets one shared rule body serve every consumer repo. Three facts run:

1. Non-vacuous floor (ObservabilityConventionTestsBase.cs:85-86): at least MinimumAlertSpecs alerts must parse out of the bicep, defaulting to 3 (ObservabilityConventionTestsBase.cs:39). ADC overrides it to five, the number of specs it declares, so a parse drift that discovered fewer would fail rather than pass vacuously (OPERATIONS.md:389-393).
2. Forward direction: every discovered alert key must have a ### ...-alert-<key> heading in OPERATIONS.md, and that heading must carry the alert's current severity as the literal text (sev N) (ObservabilityConventionTestsBase.cs:106). Re-tiering an alert in bicep without touching its runbook heading fails the build.
3. Reverse direction (ObservabilityConventionTestsBase.cs:123-127): a runbook section for an alert bicep no longer provisions is an orphan and also fails the build.

The parse window is the text between var sloAlertSpecs and resource sloAlerts (ObservabilityConventionTestsBase.cs:135-136), which in ADC's bicep is main.bicep:390 through main.bicep:482. It holds five keys and five severities, so the gate discovers five specs.

ADC's five ### sections are adc-prod-alert-failed-requests-v2 (sev 2, OPERATIONS.md:17), adc-prod-alert-server-response-time-v2 (sev 3, OPERATIONS.md:31), adc-prod-alert-dependency-failures-v2 (sev 2, OPERATIONS.md:50), adc-prod-alert-resilience-circuit-open-v2 (sev 2, OPERATIONS.md:63) and adc-prod-alert-ai-scoring-token-ceiling-v2 (sev 3, OPERATIONS.md:111). The headings carry the same -v2 suffix the provisioned rules use (main.bicep:486), and that suffix is itself load-bearing: main.bicep:484-485 warns that -v2 is part of the rule's identity in Azure, so renaming it would create a second rule beside the live one instead of updating it. The gate matches on the -alert-<key> infix (ObservabilityConventionTestsBase.cs:32), so the suffix does not affect the pairing either way.

Two of the five need a sentence each, because their remedy is not "fix the service". The circuit-open alert fires on the first breaker opening, and the breaker closes itself once a probe call succeeds, so the alert can auto-mitigate while the dependency is still sick: the runbook says to treat it as "a dependency is down", not as "something needs restarting" (OPERATIONS.md:65-70). The AI token ceiling sums the MMCA.Common.AI meter's input and output tokens over a rolling two-day window against aiScoringTokenCeiling (2,000,000 by default), and it is the only cost alert in the deployment (OPERATIONS.md:113-116). It used to sit outside the gated block; it moved into sloAlertSpecs, so its runbook moved up into the gated section with it (OPERATIONS.md:160-161).

[Rubric §31, Cost / FinOps] assesses whether cost is actively managed. Putting a token budget behind the same pairing gate as the availability SLOs means an AI spend alert cannot ship without a written answer for who reacts to it.

The operational alerts, and why their headings use four hashes

Six further alerts are provisioned outside the gated window (OPERATIONS.md:149-161): the four entries of scheduledQueryAlertSpecs (main.bicep:575-600), materialized at severity 2 with a 15-minute evaluation over a 15-minute window (main.bicep:602-613); one standalone scheduled query rule for the log-ingestion cap (main.bicep:654); and the Sev 1 gateway-availability metric alert (main.bicep:733). The section preamble explains the heading depth, and it is worth understanding rather than copying: the gate's heading regex (ObservabilityConventionTestsBase.cs:171) matches only a line that starts with exactly three hashes followed by whitespace, so a #### heading does not match it at all. That makes these sections invisible to both directions of the gate. Promoting one to ### would break the build immediately, because the reverse-direction fact would see a runbook section for an alert sloAlertSpecs does not provision and call it an orphan.

• adc-prod-alert-outbox-dead-letter (sev 2, OPERATIONS.md:163-205) fires on any AppTraces row at Error or above matching dead-lettered (main.bicep:577-579). The threshold is 0, meaning first hit rather than a rate, because every hit is an integration event permanently lost from a service's outbox. The runbook makes the reason explicit (OPERATIONS.md:165-168): the true "stuck outbox" signal is row age, which is DB-side and not queryable from Log Analytics, so this Error line has to serve as the backlog alarm as well as the loss alarm. The triage walks type-resolution failures, broker rejection and consumer-side handler failure, and warns that replay is manual; production's 300-second outbox poll (main.bicep:1746) means waiting five minutes before concluding a reset did not take.
• adc-prod-alert-sql-dependency-failures (sev 2, OPERATIONS.md:206-231), threshold 10 failed SQL dependency calls over 15 minutes (main.bicep:583). Its value over the general dependency-failures SLO is attribution: every service owns exactly one database, so a burst names a service and a database instead of "some dependency" (OPERATIONS.md:208-210). The last step (OPERATIONS.md:228-230) is the one that saves an incident: do not restart the service, because production sets DatabaseInitStrategy=Migrate and each service is the sole migrator of its own database, so a restart re-runs startup migrations against the same unreachable server and turns a read outage into a failed revision.
• adc-prod-alert-revision-activation-failed (sev 2, OPERATIONS.md:232-261) fires on the first ContainerAppSystemLogs_CL row whose Reason_s starts with "Deployment Progress Deadline Exceeded" (main.bicep:589-592), the platform's report that a revision's readiness probe never went green. It exists because that failure mode is silent from outside: the platform keeps the previous revision serving, so the Gateway availability probe stays green while production runs older code, which is how the 2026-08-29 Redis readiness regression ran unnoticed (OPERATIONS.md:234-241, main.bicep:563-568). Its deploy-side twin is the activation gate in deploy.yml:1470-1481, described in the CI/CD chapter.
• adc-prod-alert-log-ingestion-cap-reached (sev 2, OPERATIONS.md:262-295) is the alert that keeps the rest honest. The workspace carries a daily ingestion cap (5 GB/day against a ~0.4 GB/day baseline), and when the cap is reached Log Analytics stops ingesting every table until the next UTC midnight, so every other log-based alert silently evaluates empty data (OPERATIONS.md:264-268). The template records why that is a security concern and not just an outage: tripping the cap is a viable first move for an attacker (drive an error loop, then act unobserved) (main.bicep:640-644). The rule queries _LogOperation because that table is cap-exempt and keeps recording after ingestion stops (main.bicep:646-649), and it uses a 1-hour window against a 15-minute cadence so ingestion-latency skew cannot drop the once-written cap record between two buckets (main.bicep:651-653, :664-665).
• adc-prod-alert-gateway-availability (sev 1, main.bicep:733-754, OPERATIONS.md:296-318) is driven by the standard web test that pings the public Gateway /health from three Azure locations (main.bicep:700-712) with a 2-of-3 failed-location threshold (main.bicep:754). It is the only outside-in signal in the deployment, which is exactly why it is Sev 1: every other alert is reported by the app and therefore goes quiet when the app is down (OPERATIONS.md:298-299). The triage carries two facts a first responder will otherwise get wrong: /health is the Gateway's readiness endpoint and aggregates one downstream check per service, so a healthy Gateway can still fail the probe because a backend is unhealthy; and Identity, Conference and Engagement serve HTTP/2 cleartext only, so probing them without --http2-prior-knowledge reports a failure that is not there (OPERATIONS.md:310-316).

One number in that last section has drifted. OPERATIONS.md:300 still describes the synthetic probe as running at a "5-minute frequency". The web test is declared with Frequency: 900 (main.bicep:712), so each location probes every 15 minutes, and the alert's window is PT15M to match (main.bicep:747-748). The template explains the window separately from the cadence, and the window change is correctness, not cost: at Frequency: 900 a PT5M window would usually be empty, so the rule would evaluate nothing at all, while PT15M restores exactly one result per location per window, which is what failedLocationCount counts (main.bicep:742-746). The locations and the 2-of-3 threshold are unchanged; what moved is detection latency, from about 5 minutes to about 15. Read the runbook's "5-minute" as the old cadence and the bicep as current.

[Rubric §31, Cost / FinOps] assesses whether cost is actively managed. The cadence changes on the SLO rules (dated 2026-09-02 at main.bicep:502-506) and on the web test are cost decisions taken with the billing model written down beside them, and neither moved a threshold or a location. That is the distinction that keeps a cost change from quietly becoming an alerting change.

The one gap the gate cannot see

signalr-backplane-errors is the fourth scheduledQueryAlertSpecs entry (main.bicep:595-599): any Warning-or-above AppTraces line whose logger category starts with Microsoft.AspNetCore.SignalR.StackExchangeRedis, threshold 0. It exists because a backplane failure is silent by design. When the backplane loses Redis, each Notification replica keeps serving its own connections, so a push issued on one replica silently stops reaching clients held by the other, and readiness stays green on purpose, because making Redis a readiness dependency would take the whole hub down during a blip (main.bicep:567-574).

It has no runbook section, and the file says so twice (OPERATIONS.md:158-160, :394-395): the alert's own description in main.bicep:596 is the only triage pointer. Because the rule lives outside the gated window, nothing failed when it was added. That is the honest boundary of the pairing gate: it covers the five SLO specs, not the alert surface as a whole, and the governance note (OPERATIONS.md:387-395) says so, calling the non-SLO alerts "the honour system". The previous occupant of this gap, revision-activation-failed, now has its section (OPERATIONS.md:232-261), which is the honour system working, but only by hand.

Conference day

OPERATIONS.md:320-361 turns the conference-day surge from a set of az commands into one switch. The surge is declared in main.bicep as conferenceMode (main.bicep:157, default false) and driven by ONE repository variable, CONFERENCE_MODE_UNTIL (yyyy-MM-dd, UTC) (OPERATIONS.md:322-325). The deploy reads it on every run: it rejects a malformed value (deploy.yml:1435-1437) and turns the mode on while today's UTC date is not past the variable (deploy.yml:1440-1442), so the first deploy after the date reverts the surge with no one remembering to. The runbook forbids scaling by hand with az, because cost-guard.yml treats an out-of-band surge as drift and blocks deploys (OPERATIONS.md:325); cost-guard is a deploy prerequisite (deploy.yml:783-793, :1192).

What the mode changes, sized by the 2026-10-02 capacity review for about 400 concurrent attendees against a 2026 peak of about 67 (OPERATIONS.md:327-341):

• Per-client-IP limits, because a whole venue Wi-Fi is one NAT address: the gateway edge window, the gateway auth-tight policy, Identity's auth-ip limit, registrations per IP per hour and the UI host window all rise by an order of magnitude or more (OPERATIONS.md:330-334).
• Compute: Identity at 1 vCPU / 2 GiB, because every sign-in is a 600,000-iteration PBKDF2-SHA512 hash and the 2026-10-03 rehearsal saturated four 0.5 vCPU replicas at about five sign-ins a second; the other attendee-facing apps at 0.5 vCPU / 1 GiB, all five scaling to four replicas. Identity, Conference and Engagement keep a minimum of 1 because each is the sole startup migrator and seeder of its database (OPERATIONS.md:335-340).
• Databases: ADC_Conference and ADC_Engagement move online from Basic (5 DTU) to S2 (50 DTU) (OPERATIONS.md:341).

The steps (OPERATIONS.md:345-361): set the variable about a week before and run the deploy; run the dress rehearsal, conference-day-load-test.yml (manual only, and it writes to production), first in mode=seed to register 400 synthetic accounts on the reserved loadtest.adc.invalid domain (no confirmation email, never broadcast recipients), then in mode=attendees, which must pass including signed_in > 99%; and after the conference, deploy once more to revert and delete the variable. Until that reverting deploy runs, the Monday cost-guard.yml fails as the reminder. The rehearsal with the mode OFF is expected to fail on 429s; that is the baseline it proves.

[Rubric §31, Cost / FinOps] assesses whether capacity tracks need. A date-bounded switch makes the expensive footprint expire by construction instead of by memory, and the scheduled cost guard catches the one case the date cannot. [Rubric §29, Resilience & Business Continuity] is served by the same switch: it is what keeps a second front-door replica alive during the one week a replica loss would matter.

Recovery moves

OPERATIONS.md:363-385 is the fast reference: roll a bad revision back with az containerapp revision list and revision copy, follow DISASTER-RECOVERY.md for a database restore, re-run the referenced workflow when a freshness gate blocks a deploy, and revert a surge. Its freshness quick-reference (OPERATIONS.md:370-373) names three windows, and each one matches the step that enforces it: dr-freshness 8 days (deploy.yml:853), load-freshness 35 days (deploy.yml:872), cross-service-freshness 5 days (deploy.yml:897). Those numbers live in two places, so treat the workflow as the source of truth and re-check the runbook line whenever a window moves: deploy.yml enforces a fourth window the quick reference does not name, cross-browser-freshness at 10 days (deploy.yml:936), which is the drift this pairing produces whenever a gate is added and the prose is updated separately.

The two surge entries are new and worth reading as a pair. The routine revert is the conference-day step: deploy once CONFERENCE_MODE_UNTIL has passed, then delete the variable (OPERATIONS.md:374-375). The other is a deadlock breaker (OPERATIONS.md:376-385): if someone surged by hand with az and the variable is unset, cost-guard blocks every deploy and a plain re-run cannot clear it, because the deploy job needs cost-guard to pass. The way out is to set the variable to yesterday, because an expired window is the one state in which the gate lets a surged footprint through, and the deploy it allows applies main.bicep at baseline; then delete the variable and re-run cost-guard.yml to confirm.

Why there is no Aspire dashboard in production

OPERATIONS.md:400-431 closes the file by answering the question every new operator asks. The absence is a decision (ADR-098), not an omission, and it rests on three facts.

The production telemetry stream is deliberately thinned, so a full-fidelity dashboard would have nothing extra to show: main.bicep sets Telemetry__TracesSampleRatio=0.25 (main.bicep:280), an OpenTelemetry log floor of Warning so Information still reaches container stdout without billing against the workspace, Telemetry__DisableHttpClientMetrics and Telemetry__DisableRuntimeMetrics (main.bicep:317; together about 65% of AppMetrics ingestion per OPERATIONS.md:410-411), and OTEL_METRIC_EXPORT_INTERVAL=300000 against a 60-second default (main.bicep:347). The runbook's own pointer, infra/main.bicep:223-263 (OPERATIONS.md:407), predates later template growth; the settings now sit around main.bicep:280-347. The dashboard's value is live, unsampled, per-request detail, which is exactly the data production does not carry.

The durable operational surface is the workspace, not a dashboard (OPERATIONS.md:416-420): the alert rules and their action group, the SLO workbook, and KQL over ContainerAppConsoleLogs_CL, AppRequests and AppDependencies all survive a revision restart and stay queryable weeks later. And the ACA dashboard component is ephemeral and full-fidelity, which is the wrong pair for production (OPERATIONS.md:422-426): it holds its data in the running container's memory, so a restart discards it, while ingesting at exactly the fidelity the settings above were tuned to avoid.

[Rubric §31, Cost / FinOps] meets §13 here: the observability posture is shaped by ingestion cost, and the runbook records that trade-off where an operator will actually hit it rather than leaving it implicit in the bicep.

infra/SQL-MANAGED-IDENTITY.md, staged passwordless-SQL runbook

File: MMCA.ADC/infra/SQL-MANAGED-IDENTITY.md (100 lines)

What it is. The runbook for moving the four service apps from SQL-login (password) auth to Entra managed-identity auth against their per-service databases, and the written acceptance of the SQL data plane staying on public network access with no VNet.

When to run. Only when deliberately advancing that hardening. Nothing here is on the normal deploy path: all three bicep knobs are default-safe, so a deploy with the defaults changes nothing (SQL-MANAGED-IDENTITY.md:7-9, the knob table at SQL-MANAGED-IDENTITY.md:29-33).

[Rubric §11, Security] assesses credential hardening. All three stages have been run in ADC production. The template default is still false (main.bicep:36), because that is what makes a fresh environment start on password auth and each stage independently deployable, but the deployed value comes from a repository variable, not the default: deploy.yml:1313 reads vars.USE_MANAGED_IDENTITY_SQL and deploy.yml:1482-1487 rewrites the parameter to a literal JSON true when it is set. All three variables are set in ivanball/ADC (USE_MANAGED_IDENTITY_SQL, SQL_AAD_ADMIN_LOGIN and SQL_AAD_ADMIN_OID, all dated 2026-06-28), so the deployed apps use passwordless Active Directory Managed Identity connection strings. The ADC scorecard records the same activation on that date, with all four services healthy and mapped db_owner in every per-service database, which is what lifted §17 DevOps Implementation from 8 to 9. OPERATIONS.md:45-46 is therefore correct when it tells an on-call engineer that production runs passwordless SQL.

Note for a reader trying to verify this: the live value is repository configuration, not source, so it cannot be confirmed from the tree. Reading main.bicep:36 alone gives the opposite impression. ADR-061 records the staging mechanism and describes the flag as not yet flipped, which was true when it was written.

Why staged, and the three stages

The migration is staged because managed-identity auth needs a database user mapped to the identity (CREATE USER ... FROM EXTERNAL PROVIDER), which is data-plane T-SQL that must run as an Entra admin. The deploy principal holds Contributor, not SQL AAD-admin, so it cannot perform that grant. Flipping the connection strings before the grants exist takes every app offline (SQL-MANAGED-IDENTITY.md:11-18). Password auth keeps working at every step, because azureADOnlyAuthentication is deliberately not set (SQL-MANAGED-IDENTITY.md:35-37), so each stage is independently deployable and reversible.

1. Stage 1, add the Entra admin (SQL-MANAGED-IDENTITY.md:50-54). Set the SQL_AAD_ADMIN_LOGIN and SQL_AAD_ADMIN_OID repository variables; deploy.yml:1474-1481 folds them into the bicep parameter file, and main.bicep:624-633 provisions the AAD admin only when the object id is non-empty. Additive, zero app impact.
2. Stage 2, grant the identity in each database (SQL-MANAGED-IDENTITY.md:56-68). Connect to each of the four ADC_* databases as the Entra admin and run CREATE USER [adc-prod-apps-identity] FROM EXTERNAL PROVIDER plus ALTER ROLE db_owner ADD MEMBER. db_owner because each service is its own sole migrator and applies DDL at startup (SQL-MANAGED-IDENTITY.md:66-68, which also names the tighter alternative).
3. Stage 3, flip the apps (SQL-MANAGED-IDENTITY.md:70-75). Set USE_MANAGED_IDENTITY_SQL=true and deploy; main.bicep:167-169 swaps the auth segment of every connection string to Authentication=Active Directory Managed Identity with the UAMI's client id and no password. No app-code or package change is needed: Microsoft.Data.SqlClient honours the keyword and Azure.Identity is already present transitively.

Rollback is unsetting the variable and redeploying (SQL-MANAGED-IDENTITY.md:77-79): the password path is never removed in this wave, and the per-database users are inert when unused.

Accepted risk (SQL-MANAGED-IDENTITY.md:81-100). The SQL server keeps publicNetworkAccess: Enabled and the AllowAzureServices rule, because there is no VNet and moving to a private endpoint would require recreating the Container Apps environment (and therefore every container app). The compensating controls are a TLS 1.2 minimum (main.bicep:604), Key Vault secrets reached through the managed identity, and, with stages 1 to 3 complete, no shared SQL password in the data path at all, so network reachability no longer implies credential exposure.

infra/POST-CUTOVER-atldevcon-downgrade.md, archive downgrade and drop runbook

File: MMCA.ADC/infra/POST-CUTOVER-atldevcon-downgrade.md (117 lines)

What it is. The record of what happened to the legacy AtlDevCon database after the database-per-service cutover, in three acts. The first act is the S0 to Basic downgrade that was the third and final commit of the rollout. The second, on 2026-09-02, is the archive-and-drop that retired the database entirely. The third, on 2026-10-03, is the deletion of the archive itself. The file opens with a status banner (POST-CUTOVER-atldevcon-downgrade.md:3-9) that tells you which act is current: AtlDevCon no longer exists, the bacpac it was exported to was itself deleted on 2026-10-03, and everything above the "Final state" section is history whose "never delete this resource" instruction has been superseded.

When to consult. When someone asks for the pre-cutover data (the answer is that it no longer exists outside the live databases), when you are rebuilding the environment after a disaster and want to know what the archive was and why there is none, or when you meet the "NEVER delete" comment quoted in an older document and need to know it was retired deliberately. Nothing in this runbook is a step on the normal deploy path any more.

[Rubric §31, Cost/FinOps] assesses whether cost is actively managed and right-sized. This file is that principle applied twice, with a measurement between the two applications. The first pass downgraded an idle database from S0 to Basic. The second pass retired it altogether once the numbers were in: 32 MB of data and 0 DTU for the whole summer while still billing as a Basic database (POST-CUTOVER-atldevcon-downgrade.md:81-83). The reasoning fits in one sentence: a "data must never be lost" constraint was satisfied just as well by a bacpac in blob storage, at a small fraction of the cost.

[Rubric §8, Data Architecture] assesses data lifecycle and migration hygiene. Retiring the data estate's last legacy member happened in deliberate, recorded steps: an export was verified present and non-empty before the drop, and the later deletion of that export is recorded with its date, its reason and its consequence rather than performed silently (POST-CUTOVER-atldevcon-downgrade.md:103-110).

[Rubric §30, Compliance & Privacy] assesses whether personal data is kept only as long as it has a purpose. The archive was deleted as personal data with no remaining purpose, by the owner's decision, and the runbook states plainly that blob soft delete and versioning were off, so no copy remains (POST-CUTOVER-atldevcon-downgrade.md:105-108). Writing down that the deletion is irreversible is what makes it auditable.

Walkthrough

Act one, the downgrade (POST-CUTOVER-atldevcon-downgrade.md:11-77). Kept as history, and still worth reading for the shape of a safe SKU change:

1. Prerequisites (:21-41). Confirm the flip is live and healthy (all four adc-prod-* apps serving from their ADC_* databases, no writes to AtlDevCon for at least 24 h, its outbox drained to a ProcessedOn IS NULL count of 0), confirm the database fits the 2 GB Basic cap with az sql db list-usages and leave it at S0 if it does not (:27-32), and export a .bacpac first because Basic drops PITR retention from 35 days to 7 (:34-41).
2. The Bicep change (:43-65). Replace sku and maxSizeBytes only. Note the inline supersession marker at :47-48: the original instruction was "keep the resource, never remove it", and the file flags that line as overtaken by the drop rather than quietly rewriting it, which is what lets an operator holding an older copy recognize which version they have.
3. Why commit 3 was separate (:18-19). If main.bicep had carried the Basic SKU when commit 2 deployed, AtlDevCon would have been downgraded before the new databases were proven. Separating the commits eliminated that risk.
4. Deploy, verify, roll back (:67-77). Merge and let the normal deploy.yml run apply it (Incremental mode changes only the SKU, data untouched), verify with az sql db show that the SKU reads Basic and the cap is 2147483648, and revert the commit to return to S0 if needed. Rolling back the SKU was always independent of rolling back the app flip.

Act two, archive and drop (POST-CUTOVER-atldevcon-downgrade.md:79-101). Three steps, in this order, and the order is the whole design (:85-101):

1. infra/main.bicep stopped declaring the sqlDatabase resource, and that change deployed. The database was still there afterwards, because Incremental mode does not delete a resource just because it left the template. That property is what made the sequence safe rather than a race.
2. az sql db export wrote sql-archive/AtlDevCon-20260902.bacpac into storage account adcprodstpys4way4uzb3g.
3. The export was verified present and non-empty in the blob container, and only then was the database dropped by hand with az sql db delete.

Act three, archive deleted (:103-110). The bacpac was deleted permanently on 2026-10-03. There is no longer any rollback path to the pre-cutover AtlDevCon data; its contents had already been copied into the live per-service databases at cutover, and the four live ADC_* databases keep their own PITR plus LTR. The empty sql-archive container remains. The az sql db import restore this section used to carry is gone with it, which is also what unblocked turning off storage Shared Key access (see the DR section above).

"Not ours": the atldevcon SQL server (:112-117). A SQL server named atldevcon in westus2 lives in the same shared resource group. It predates MMCA, belongs to something else, and has never hosted an ADC database. In a shared resource group, a name collision like this is a real operational hazard, and writing it down is the mitigation.

play-store-capture.ps1, Android screenshot capture

File: MMCA.ADC/scripts/play-store-capture.ps1 (147 lines)

What it is. A PowerShell 7 script (play-store-capture.ps1:1) that captures a screenshot from an attached Android device or emulator via adb screencap and saves it as a deterministic-filename PNG under store-assets/play-store/raw/.

When to run. When preparing or refreshing Google Play Store screenshots. Run once per slot (for example 01-home, 02-sessions) on a device or emulator that is showing the correct screen. Eight slots are defined in the companion compose script's lineup.

[Rubric §30, Compliance/Privacy] assesses whether the app store presence is maintained. These scripts are the mechanism for maintaining Play Store assets; without them the screenshots drift from the current UI. The raw/ directory is gitignored because a capture can contain the signed-in test user's data; only the composed screenshots/ set is committed (store-assets/play-store/README.md:19-20).

Walkthrough

Slug-based filenames (play-store-capture.ps1:47-51, written at :106). Raw captures are saved as <slug>.png under store-assets/play-store/raw/, with the directory created on first use (play-store-capture.ps1:48-49). Deterministic names mean the compose script can look them up by slug key without a manifest file.

-List switch (play-store-capture.ps1:53-65). Lists the slots already captured, with file size and timestamp: a quick sanity check before a composing session.

adb resolution (play-store-capture.ps1:67-75). Prefers adb on PATH, then falls back to the well-known SDK path (C:\Program Files (x86)\Android\android-sdk\platform-tools\adb.exe), and throws an actionable error when neither exists (play-store-capture.ps1:73-74).

exec-out not shell screencap (play-store-capture.ps1:104-110). Uses adb exec-out to stream the PNG binary straight into the output file. Plain adb shell screencap on Windows applies CRLF translation to the binary stream, corrupting the PNG. This is the kind of platform detail that is invisible until every capture is subtly broken, which is why the comment sits directly above the call. A capture smaller than 1024 bytes is deleted and reported as an error rather than kept (play-store-capture.ps1:121-125), because at that size the device was asleep or locked.

Dimension read from IHDR (play-store-capture.ps1:127-147). Reads the PNG IHDR chunk (bytes 16-19 width, 20-23 height, big-endian, play-store-capture.ps1:127-128) to print the captured resolution. A landscape capture (width greater than height) emits a warning, because Play Store phone screenshots must be portrait.

play-store-compose.ps1, Play Store screenshot compositor

File: MMCA.ADC/scripts/play-store-compose.ps1 (187 lines)

What it is. A PowerShell 7 script that reads raw captures from store-assets/play-store/raw/, wraps each into a 1080x1920 branded canvas (play-store-compose.ps1:67-68), overlays a caption and subtitle from a hard-coded lineup, and writes the finished PNG to store-assets/play-store/screenshots/. The composed images satisfy Play Console's aspect-ratio requirement: Pixel emulators capture at 1080x2400 (9:20), which Play Console rejects as too tall, and the script guarantees a compliant 1080x1920 (9:16) result (play-store-compose.ps1:11-13).

When to run. After play-store-capture.ps1 has captured all needed slots. Run with no -Slug to compose the full set, or -Slug <slug> to recompose one slot (the filter, and its clear error when no raw capture matches, is at play-store-compose.ps1:82-85). Run with -NoCaption for a plain brand-framed variant (a feature graphic or tablet screenshots, for example), which also shrinks the reserved caption band from 280 px to 80 px (play-store-compose.ps1:72).

[Rubric §30, Compliance/Privacy] assesses app store compliance. Play Console has strict aspect-ratio rules for phone screenshots; this script enforces compliance mechanically rather than relying on manual cropping.

Walkthrough

Lineup (play-store-compose.ps1:52-61). Eight slots are defined inline with slug, caption (large bold white text) and subtitle (smaller cyan text). The comment at play-store-compose.ps1:51 notes that the lineup must be kept in sync with store-assets/play-store/README.md, whose table (store-assets/play-store/README.md:42-51) currently matches all eight slugs, captions and subtitles exactly. A missing lineup entry does not fail the run: the slot renders without a caption and a warning is written (play-store-compose.ps1:158-160).

Brand colors (play-store-compose.ps1:69-71). Three colors: brandTeal (#0D7377), brandCyan (#14FFEC) and brandTealDark (#094F52). The canvas uses a vertical gradient from teal to dark teal for depth (play-store-compose.ps1:95-98).

System.Drawing.Common assembly (play-store-compose.ps1:39-42). The script loads System.Drawing.Common (the PowerShell 7 path) inside a try, falling back to the desktop-FX System.Drawing alias on Windows PowerShell 5.1.

Fit-inside scaling (play-store-compose.ps1:130-138). Each raw capture is scaled to fit inside the available image area (imageMaxW by imageMaxH, computed from the canvas minus the caption band and the footer at play-store-compose.ps1:74-75), preserving aspect ratio, then centered horizontally and vertically. Captures taller than 9:16 (the 9:20 Pixel emulator case) are letterboxed on the teal background rather than cropped, so nothing in the UI is lost.

Soft shadow and cyan border (play-store-compose.ps1:140-152). A black rectangle offset 10 px right and 14 px down at alpha 80 of 255 produces a drop shadow; a 2-px cyan border makes the screenshot pop against the teal background.

Output (play-store-compose.ps1:175). Each composed PNG is saved as store-assets/play-store/screenshots/<slug>.png, and every GDI+ object is released in nested finally blocks (play-store-compose.ps1:176-182) so a long composing run does not leak handles. Upload the results directly to Play Console, Main store listing, Phone screenshots (play-store-compose.ps1:187).

Docs/MobileReleaseRunbook.md, store-submission runbook

File: MMCA.ADC/Docs/MobileReleaseRunbook.md (205 lines)

What it is. The manual, credential-holding steps around a store submission that code and CI cannot perform, each tagged with when it must happen relative to the submission (MobileReleaseRunbook.md:3-5). Its companion is Docs/DeviceTestChecklist.md (178 lines), the physical-device pass to re-run after any behavior-affecting bump.

When to run. Before a Play or TestFlight upload. The eight sections are independent; read the one that matches the step you are on.

[Rubric §17, DevOps & Deployment] assesses whether the release path is documented end to end. The mobile head is the part of ADC that CI cannot fully automate (signing keys, provisioning profiles, store consoles), so the runbook is the substitute for a pipeline.

The load-bearing items

Android App Links fingerprint (MobileReleaseRunbook.md:7-59). /.well-known/assetlinks.json is served by the shared MapAppAssociationEndpoints mapper in MMCA.Common.API, wired in MMCA.ADC.UI.Web/Program.cs from the AppAssociation configuration section; bicep does not inject that section and appsettings.Production.json does not override it, so the base appsettings.json values are verbatim what production serves (MobileReleaseRunbook.md:13-18). Android verifies a link only when both package_name and sha256_cert_fingerprints match the installed app. Two traps: the Release android property group overrides the package id to ivanball.AtlDevCon (MMCA.ADC.UI.csproj:58) while Debug installs as com.ivanball.atldevcon (MMCA.ADC.UI.csproj:29), and the checked-in fingerprint is still a placeholder that must be replaced with the Play App Signing certificate fingerprint, not the local upload keystore's, which Google discards when it re-signs (MobileReleaseRunbook.md:32-34; ADR-043 records the same open item). The fingerprint belongs in the repo rather than Key Vault because it is public by design: it is served to anyone who requests the document (MobileReleaseRunbook.md:39-40).

iOS associated domains (MobileReleaseRunbook.md:61-75). Entitlements.plist requests com.apple.developer.associated-domains and is wired in through CodesignEntitlements (MMCA.ADC.UI.csproj:81). The existing "AtlDevCon App Store" provisioning profile does not carry that capability, so the next Release build fails signing until the capability is enabled and the profile regenerated under the identical name (the csproj pins CodesignProvision, MMCA.ADC.UI.csproj:90). The AASA document must be live on the prod UI host before App Review, because Apple's CDN fetches it at submission (MobileReleaseRunbook.md:72-75).

Release signing password (MobileReleaseRunbook.md:77-81). Keystore passwords are not in the csproj: AndroidSigningKeyPass and AndroidSigningStorePass read the ADC_ANDROID_SIGNING_PASSWORD environment variable (MMCA.ADC.UI.csproj:73-74), which must be set in the shell before any local Release android build.

Native push and the avatar grant, two Bicep switches pointing in opposite directions. Both are worth reading against the template rather than the prose:

• Native push is wired and enabled by default: nativePushEnabled defaults to true (main.bicep:119-120) and deployNotificationHub defaults to true (main.bicep:122-123). What the second flag gates is only the wiring, the Key Vault connection-string secret plus the notification app's secret and env refs (main.bicep:960-964, :1523, :1591-1595). The namespace, hub and auth rule are never deployed by this template: they are declared existing (main.bicep:754-769) because ARM PUTs on this namespace never reach a terminal state, hit on create in 2026-07-11 and again on update in 2026-08-24 (main.bicep:746-753). They have to be provisioned out of band through section 5 of the runbook before the flag can be true. Read that section from the bottom up: the opening paragraph (MobileReleaseRunbook.md:94-99) still says the hub is not provisioned and describes the guard as deployNotificationHub=false, but the progress note that follows (MobileReleaseRunbook.md:101-105) records the current defaults, and the remaining manual work is the out-of-band hub creation, the FCM v1 service-account credential and the APNs key (MobileReleaseRunbook.md:107-121). Registration works on the connection string alone; only sends need those credentials.
• Avatar storage still needs a one-time manual role grant, because the deploy identity deliberately lacks Microsoft.Authorization/roleAssignments/write (main.bicep:125-126, grantAvatarStorageRole default false, the guarded resource at main.bicep:831-839). Until the grant is applied, avatar uploads fail cleanly with FileStorage.UploadFailed and nothing else breaks (MobileReleaseRunbook.md:123-146, which includes the az rest form because az role assignment create misreports on this tenant). One detail the template adds and the runbook does not: the grant is scoped to the storage account, so the same assignment also covers the Data Protection key-ring container (main.bicep:825-826).

Play target API level (MobileReleaseRunbook.md:155-205). A recurring annual deadline: from 2026-08-31 an update must target API 36 (Android 16) or higher. The level is pinned in the csproj (MMCA.ADC.UI.csproj:54, TargetPlatformVersion 36.0) precisely so a build machine one Android workload behind fails outright instead of quietly producing a rejectable AAB (MobileReleaseRunbook.md:171-176). Raising it is a behavior change (API 35+ forces edge-to-edge layout), so the device checklist is re-run before rollout, and the runbook prescribes verifying the merged manifest before every upload (MobileReleaseRunbook.md:183-192).

The operational artifacts in full context

The surviving artifacts form one story that starts at bootstrap and ends at day-2 triage:

The AtlDevCon database is the thread that runs through the database half of that table, and that thread now ends: it was the source of data truth during the copy, then the rollback path after the flip, then a Basic-tier archive, and since 2026-09-02 a bacpac blob rather than a database (main.bicep:635-647, POST-CUTOVER-atldevcon-downgrade.md:79-121). Three practical consequences follow for an operator. The four live ADC_* databases are the entire estate covered by PITR and LTR (main.bicep:681-685). The weekly drill rotates over exactly those four and cannot target anything else (dr-drill.yml:21-25, :79-82). And recovering pre-cutover data is an az sql db import into a new database name, not a point-in-time restore.

Cross-links:

• IaC chapter: infra/main.bicep provisions the four ADC_* databases, the LTR policies, the SLO alerts and workbook, and the Service Bus namespace.
• CI/CD chapter: deploy.yml carries the revision-activation and smoke gates, the rollback, and the four recency gates (dr-freshness, load-freshness, cross-service-freshness, cross-browser-freshness, deploy.yml:1219).
• ADR-006, the decision to adopt database-per-service, its trade-offs, and the CrossDataSourceDegradeConvention that removes cross-database FKs.
• ADR-009, the resilience and recovery objectives framework, including the requirement that DISASTER-RECOVERY.md exists and that the drill-result table is filled.
• ADR-062, SLO alerting as code and the alert-to-runbook pairing gate that infra/OPERATIONS.md satisfies.
• ADR-064, deploy preconditions as proof of recency, which is what turns the weekly drill into a production gate.
• ADR-098, Aspire for orchestration and not for production dashboards, which OPERATIONS.md:176-207 operationalizes.
• ADR-003, the outbox pattern that the per-service databases each own independently and that the outbox dead-letter alert watches.

Rubric tag summary

Not determinable from source

• Whether the newest weekly drill is green: the drill ledger is maintained through 2026-08-10 (DISASTER-RECOVERY.md:210-217), but dr-drill.yml runs every Monday and a run reaches the ledger only when an operator pastes the printed row back into DISASTER-RECOVERY.md. Any drill newer than the last ledger row exists only in the Actions history, which is exactly what dr-freshness queries (deploy.yml:895-899, which reads MMCA.Common/.github/actions/freshness-gate/action.yml:184-187); it cannot be read from the repository.
• The live values of the three SQL managed-identity repository variables: main.bicep:36 shows the safe default (false) and deploy.yml:1313 shows that the deployed value comes from vars.USE_MANAGED_IDENTITY_SQL. The variable's current value is repository configuration, not source. Check Settings, then Secrets and variables, then Actions.
• Whether the archive bacpac is still present and readable: the blob path and storage account are recorded (POST-CUTOVER-atldevcon-downgrade.md:103-121, main.bicep:638-639) and the export was verified at the time of the drop (POST-CUTOVER-atldevcon-downgrade.md:97-101), but nothing in the repository can attest to the blob's current state. It is now the only copy of the pre-cutover data, so an az storage blob show against that container is worth running before anyone relies on it.
• Whether the Notification Hub namespace actually exists in Azure: main.bicep:754-769 declares the namespace, hub and auth rule as existing, so the template asserts nothing about their presence. With deployNotificationHub defaulting to true (main.bicep:123), a deploy fails to resolve listKeys() if they were never provisioned out of band. Only an az query against the subscription can settle it.
• Azure list prices: the DTU counts and the cadences are in source (main.bicep:669-673 sets Basic capacity 5 for each per-service database), and the template quotes measured monthly figures for the alert rules and the web test (main.bicep:345-349, :464-469). Those figures are point-in-time measurements on this subscription, not a price list. Check Azure pricing before quoting a saving.