Guides & specifications

Resilience & Business Continuity (rubric §29)

MMCA.Common is a library, so it cannot operate a deployment — restores, RTO/RPO, and SLO alerting are executed in the consumer apps' IaC (e.g. MMCA.ADC's infra/DISASTER-RECOVERY.md, scripts/dr-restore-drill.ps1, dr-drill.yml, and the SLO metric-alerts in infra/main.bicep). What the framework can do is (1) ship the failure-isolation and graceful-startup mechanisms, tested centrally, (2) run a central, headless restore-drill smoke against an ephemeral database that exercises the backup → catastrophic-loss → restore → verify cycle and records a baseline restore time, so the recovery procedure itself is demonstrated in CI rather than only described, and (3) give consumers a baseline SLO/error-budget template and a restore-drill runbook so the recovery story is defined once and adopted, not reinvented per app. This complements ADRs/009-resilience-and-recovery-objectives.md (the decision); this file is the operational reference.

What the framework provides (and verifies in-repo)

Failure isolation, graceful degradation, graceful startup, and the restore procedure itself are therefore demonstrated and tested centrally — the framework drills backup→restore against an ephemeral DB and records a baseline restore time (see below). The gaps a library structurally cannot fill — production RTO/RPO against real cloud backups and measured production SLOs — remain the consumer's, with the templates below.

In-repo restore-drill baseline (measured, automatically gated)

DatabaseRestoreDrillTests runs on every CI build (it lives in the in-solution unit tier, so the recovery procedure is enforced automatically, not just described): it seeds a 500-row table, backs it up, deletes all rows (the simulated disaster), restores from the backup, and asserts every row returns byte-for-byte. The measured restore time is emitted to test output (Restore RTO (measured): … ms); the assertion ceiling is a deliberately generous 30 s hang-detector, not a performance gate (CI runners vary, so a tight latency assertion would be flaky; the drill proves correctness of recovery, and the measured time is recorded as an informational baseline).

Recorded baseline (framework's own measured restore objective, distinct from the consumer template below; this is the in-repo half §29 owns):

This is the framework's own analog of the consumer cloud drill below, proving the procedure and a measured baseline centrally; the consumer drill proves it against production-grade backups and real cloud RTO targets (orders of magnitude larger, hence the per-app template below rather than a shared number).

Baseline SLO / error-budget template (consumers fill in)

Adopt and tune per app; ADC's filled-in version lives in infra/DISASTER-RECOVERY.md + the SLO metric-alerts in infra/main.bicep.

Define RTO/RPO per service (ADC's worked example):

A conscious, documented acceptance of single-region risk is a valid §29 posture — state it explicitly (as ADC does) rather than leaving it implicit.

Restore-drill runbook (reference)

The only evidence backups actually restore is a periodic drill: restore a throwaway copy, confirm it comes back Online, record the measured restore time, then delete the copy. The live databases are never touched. Run it per release train and after any backup/retention change.

# PITR restore of a throwaway COPY (Azure SQL example — adapt to your store)
          az sql db restore -g <rg> -s <sqlServer> -n <Db> --dest-name <Db>-drill --time "<recent-utc>"
          # verify status Online (and a row/table spot-check for a deeper check), record elapsed minutes, then:
          az sql db delete -g <rg> -s <sqlServer> -n <Db>-drill --yes

Wire this as a manual/scheduled workflow + script (worked example: MMCA.ADC's .github/workflows/dr-drill.yml + scripts/dr-restore-drill.ps1, which restore a copy, measure RTO, verify Online, clean up, and emit a drill-result row). Record each run: