Guides & specifications

Resilience & Business Continuity (rubric §29)

MMCA.Common is a library, so it cannot operate a deployment: restores, RTO/RPO, and SLO alerting are executed in the consumer apps' IaC (e.g. MMCA.ADC's infra/DISASTER-RECOVERY.md, scripts/dr-restore-drill.ps1, dr-drill.yml, and the SLO metric-alerts in infra/main.bicep). What the framework can do is (1) ship the failure-isolation and graceful-startup mechanisms, tested centrally, (2) run a central, headless restore-drill smoke against an ephemeral database that exercises the backup → catastrophic-loss → restore → verify cycle and records a baseline restore time, so the recovery procedure itself is demonstrated in CI rather than only described, and (3) give consumers a baseline SLO/error-budget template and a restore-drill runbook so the recovery story is defined once and adopted, not reinvented per app. This complements ADR-009 (the decision); this file is the operational reference.

What the framework provides (and verifies in-repo)

Failure isolation, graceful degradation, graceful startup, and the restore procedure itself are therefore demonstrated and tested centrally: the framework drills backup→restore against an ephemeral DB and records a baseline restore time (see below). The gaps a library structurally cannot fill (production RTO/RPO against real cloud backups and measured production SLOs) remain the consumer's, with the templates below.

In-repo restore-drill baseline (measured, automatically gated)

DatabaseRestoreDrillTests runs on every CI build (it lives in the in-solution unit tier, so the recovery procedure is enforced automatically, not just described): it seeds a 500-row table, backs it up, deletes all rows (the simulated disaster), restores from the backup, and asserts every row returns byte-for-byte. The measured restore time is emitted to test output (Restore RTO (measured): … ms); the assertion ceiling is a deliberately generous 30 s hang-detector, not a performance gate (CI runners vary, so a tight latency assertion would be flaky; the drill proves correctness of recovery, and the measured time is recorded as an informational baseline).

Recorded baseline (framework's own measured restore objective, distinct from the consumer template below; this is the in-repo half §29 owns):

This is the framework's own analog of the consumer cloud drill below, proving the procedure and a measured baseline centrally; the consumer drill proves it against production-grade backups and real cloud RTO targets (orders of magnitude larger, hence the per-app template below rather than a shared number).

Caching is fail-open (configurable, never load-bearing)

The cache is an optimization, never the system of record, so nothing about it can turn a cache problem into a failed request. A miss, an unreachable cache, or a populate that outlives its lock timeout all degrade the same way: the request falls through to the real handler, runs uncached, and still answers correctly. There is no stale-serving tier (nothing like FusionCache's fail-safe, which keeps returning an expired entry while the backing store is unreachable). That absence is the deliberate half of the tradeoff, not a gap.

CacheSettings (Source/Core/MMCA.Common.Infrastructure/Settings/CacheSettings.cs, bound from the Cache section) makes the policy configurable per host without changing that posture. The section is optional: every default is the same value CacheOptions states in code, so a host that configures nothing gets the framework policy.

Why fail-open rather than stale-serve:

• Correctness is not a framework-level tradeoff to make silently. How stale a value may be is a per-read business judgment, and a framework that served known-expired entries during an outage would change what every consumer's queries return at the worst possible moment. Falling through to the handler is the one behavior that is correct for every caller.
• A cache outage degrades to load, not to errors. The database sees the traffic it would have seen with no cache configured, which is a capacity question the mechanisms above (timeouts, retries, circuit breakers, rate limiting) already answer, rather than a correctness one.
• The pathological case has its own bounded knob. PopulateLockTimeout is stampede protection by default (waiters block until the one request holding the lock has populated the entry). A finite value trades some of that protection for a latency bound: once it elapses the waiter proceeds uncached instead of queueing behind a slow populate or a wedged handler. The cost is several concurrent executions of the same query, which is the cheaper failure of the two.

The decisions behind this behavior are ADR-026 (caching strategy) and ADR-077 (the HybridCache substrate); this section is the availability posture they imply.

Baseline SLO / error-budget template (consumers fill in)

Adopt and tune per app; ADC's filled-in version lives in infra/DISASTER-RECOVERY.md + the SLO metric-alerts in infra/main.bicep.

Define RTO/RPO per service (ADC's worked example):

A conscious, documented acceptance of single-region risk is a valid §29 posture: state it explicitly (as ADC does) rather than leaving it implicit.

Restore-drill runbook (reference)

The only evidence backups actually restore is a periodic drill: restore a throwaway copy, confirm it comes back Online, record the measured restore time, then delete the copy. The live databases are never touched. Run it per release train and after any backup/retention change.

# PITR restore of a throwaway COPY (Azure SQL example: adapt to your store)
az sql db restore -g <rg> -s <sqlServer> -n <Db> --dest-name <Db>-drill --time "<recent-utc>"
# verify status Online (and a row/table spot-check for a deeper check), record elapsed minutes, then:
az sql db delete -g <rg> -s <sqlServer> -n <Db>-drill --yes

Wire this as a manual/scheduled workflow + script (worked example: MMCA.ADC's .github/workflows/dr-drill.yml + scripts/dr-restore-drill.ps1, which restore a copy, measure RTO, verify Online, clean up, and emit a drill-result row). Record each run: