Ivan Ball-llovera

↑↓ to navigate Enter to open "…" all these words ANDOR to combine

Architecture Decision Record

ADR-058: Runtime Conformance Suites Shipped as a Package

Status

Accepted (2026-07-28; revised 2026-08-14, 2026-08-18, 2026-08-23, and 2026-09-03).

Context

ADR-015 turned the architecture invariants into build-gating tests, and drew its own boundary explicitly: the fitness suite asserts "structure / registration, not runtime behavior" (015-architecture-fitness-functions.md:48). NetArchTest can prove a controller lives in the right assembly and that an outbound client wires resilience. It cannot prove that a booted host answers a bad page number with an RFC 9457 problem document, emits HSTS on a liveness probe, drains within a bounded stop, serves the OpenAPI document it claims to serve, answers two API versions, or nests the ADR-014 decorators in the documented order. Those are runtime contracts of the framework, and every one of them is only true if the consuming host wired it correctly: the framework ships the middleware, the controller base, and the decorator registrations, but the host composes them.

The gap is visible in the ADR set. ADR-046 names exactly one of these checks in passing (a "shared fitness contract" for versioning, 046-http-api-versioning.md:78), and ADR-013 defines the RFC 9457 edge contract that another one guards, but neither decides the layer itself. The runtime-conformance tier has been referenced repeatedly and never recorded on its own.

Decision

Ship the runtime conformance suites in the MMCA.Common.Testing package as abstract behavioral bases that each consuming host subclasses, and run every one of them against a host that was actually booted.

  • Seven contract bases, one per runtime contract, all sitting under Source/Hosting/MMCA.Common.Testing/Conformance/ in namespace MMCA.Common.Testing.Conformance, so every bare TestsBase.cs name below resolves inside that folder; the fixtures they boot through sit beside it in Fixtures/. ProblemDetailsContractTestsBase<TFixture> (Source/Hosting/MMCA.Common.Testing/Conformance/ProblemDetailsContractTestsBase.cs:22) asserts the two error-shaping paths, ASP.NET Core model validation (400, application/problem+json with type/traceId/errors, ProblemDetailsContractTestsBase.cs:30) and the framework's HandleFailure Result mapping (404, ProblemDetailsContractTestsBase.cs:42), against the shared shape check AssertProblemDetailsShapeAsync (ProblemDetailsContractTestsBase.cs:68). OpenApiContractTestsBase<TFixture> (OpenApiContractTestsBase.cs:22) asserts the live /openapi/v1.json document is OpenAPI 3.x and still describes the pinned public resources (OpenApiContractTestsBase.cs:53, OpenApiContractTestsBase.cs:68). ServiceInfoVersioningContractTestsBase<TFixture> (ServiceInfoVersioningContractTestsBase.cs:20) drives /ServiceInfo at api-version: 1.0 and 2.0 and checks the deprecated/supported reporting headers (ServiceInfoVersioningContractTestsBase.cs:39, :55). SecurityHeadersTestsBase (SecurityHeadersTestsBase.cs:16) probes /alive and pins six response headers, including Content-Security-Policy: frame-ancestors 'none' and an HSTS max-age (SecurityHeadersTestsBase.cs:29-35). GracefulShutdownTestsBase<TEntryPoint> (GracefulShutdownTestsBase.cs:25) calls a real IHost.StopAsync under a bounded token and asserts ApplicationStopping then ApplicationStopped fired (GracefulShutdownTestsBase.cs:57, :59, :61). MmcaGatewayHardeningTestsBase<TEntryPoint> (MmcaGatewayHardeningTestsBase.cs:39) drives a booted gateway through eight edge gates: the per-client-IP rate limiter and its bypass list, the tighter named policy on the credential route, a correlation id generated when the caller supplies none and echoed when it does, one readiness check per downstream service, an active health probe on every cluster, and partitioning by the forwarded client IP rather than the proxy IP (MmcaGatewayHardeningTestsBase.cs:126, :148, :184, :211, :229, :246, :274, :310). DecoratorPipelineOrderTestsBase<TCommand, TCommandResult, TQuery, TQueryResult> (DecoratorPipelineOrderTestsBase.cs:38) asserts the ADR-014 nesting.
  • The host is really booted; nothing is inferred from registrations. Three of the six booted-host suites (problem details, OpenAPI, versioning) reach the host through IIntegrationTestFixture (Source/Hosting/MMCA.Common.Testing/Fixtures/IIntegrationTestFixture.cs:8) and IntegrationTestBase<TFixture> (Fixtures/IntegrationTestBase.cs:13), whose Client comes from the fixture's WebApplicationFactory. The other three take a factory directly: security headers through its own abstract CreateClient() (SecurityHeadersTestsBase.cs:42), graceful shutdown and gateway hardening through a factory the subclass supplies (GracefulShutdownTestsBase.cs:32, MmcaGatewayHardeningTestsBase.cs:61). Two boot paths ship. Service hosts use SqlServerIntegrationTestFixtureBase<TEntryPoint> (Fixtures/SqlServerIntegrationTestFixtureBase.cs:27): it creates a GUID-named throwaway SQL Server database, pushes the connection string and ASPNETCORE_ENVIRONMENT=Testing as process environment variables before the host is built (SqlServerIntegrationTestFixtureBase.cs:75, :76), lets the host's own DatabaseInitStrategy=Migrate apply the schema when the first client forces the build (SqlServerIntegrationTestFixtureBase.cs:83), resets data between tests with Respawn (SqlServerIntegrationTestFixtureBase.cs:99), and drops the database on disposal (SqlServerIntegrationTestFixtureBase.cs:127). Database-free hosts use ProductionHostApplicationFactory<TEntryPoint> (Fixtures/ProductionHostApplicationFactory.cs:23), which pins UseEnvironment("Production") (ProductionHostApplicationFactory.cs:37) so the production-only branches (restrictive CORS, HSTS emission) are the ones under test, and captures the started IHost (ProductionHostApplicationFactory.cs:30) because StopAsync is not reachable through the WebApplicationFactory surface.
  • The consumer supplies its host and its host-specific facts, nothing else. The abstract surface is deliberately small: two probe requests for problem details (ProblemDetailsContractTestsBase.cs:55, :61), a route-count floor and a pinned resource list for OpenAPI (OpenApiContractTestsBase.cs:38, :51), a client factory for security headers (SecurityHeadersTestsBase.cs:42), a service-collection configurator for the decorator pipeline (DecoratorPipelineOrderTestsBase.cs:46), the booted factory plus the host's own route-table facts (the permit limit, a limited path, the downstream service names) for gateway hardening (MmcaGatewayHardeningTestsBase.cs:61, :68, :71, :77), and for versioning and graceful shutdown nothing at all beyond the fixture or entry point. The two shutdown subclasses are one-line declarations with no body (MMCA.Store/Tests/Hosts/MMCA.Store.Gateway.Tests/GracefulShutdownTests.cs:9, MMCA.ADC/Tests/Hosts/MMCA.ADC.Gateway.Tests/GracefulShutdownTests.cs:9), and so is the ADC versioning subclass (MMCA.ADC/Tests/Integration/MMCA.ADC.Conference.IntegrationTests/Contract/ApiVersioningTests.cs:14).
  • The decorator suite is the one non-HTTP conformance check, and it inspects the object graph, not the registration list. It builds a ServiceCollection, runs the repo's own real registration sequence through the subclass, resolves the closed handler interface from the built provider, then walks outermost to innermost by reading each decorator's private inner-handler field (DecoratorPipelineOrderTestsBase.cs:105) and compares the names against the ADR-014 order (DecoratorPipelineOrderTestsBase.cs:49, :61). That is what makes it a runtime check: Scrutor TryDecorate applies decorators in reverse registration order, so a reordered AddApplicationDecorators() or a module scan that ran after it changes the constructed pipeline while every registration still exists.
  • No committed snapshots. The OpenAPI guard asserts against the live document rather than a checked-in file (OpenApiContractTestsBase.cs:16), so a new controller can never leave a stale snapshot behind, and the assertions are deliberately coarse: a path-count floor plus presence (not exact casing) of the pinned resources (OpenApiContractTestsBase.cs:79).
  • Hosts extend the base where they have more to prove. ADC Conference adds a 412 stale-precondition test on top of the inherited 400/404 facts, driving two editors through the same If-Match tag and reusing the inherited shape assertion (MMCA.ADC/Tests/Integration/MMCA.ADC.Conference.IntegrationTests/Contract/ProblemDetailsContractTests.cs:40, :69).

Adoption today is real but partial, and uneven per suite. The OpenAPI guard is the only one with full coverage of the extracted REST hosts: all four ADC services (MMCA.ADC/Tests/Integration/MMCA.ADC.Conference.IntegrationTests/Contract/OpenApiContractTests.cs:14, MMCA.ADC.Engagement.IntegrationTests/Contract/OpenApiContractTests.cs:14, MMCA.ADC.Identity.IntegrationTests/Contract/OpenApiContractTests.cs:15, MMCA.ADC.Notification.IntegrationTests/Contract/OpenApiContractTests.cs:16) and all three Store services (MMCA.Store/Tests/Integration/MMCA.Store.Catalog.IntegrationTests/Contract/OpenApiContractTests.cs:14, MMCA.Store.Identity.IntegrationTests/Contract/OpenApiContractTests.cs:14, MMCA.Store.Sales.IntegrationTests/Contract/OpenApiContractTests.cs:14). The problem-details guard covers all three Store services (MMCA.Store/Tests/Integration/MMCA.Store.Catalog.IntegrationTests/Contract/ProblemDetailsContractTests.cs:19, MMCA.Store.Identity.IntegrationTests/Contract/ProblemDetailsContractTests.cs:15, MMCA.Store.Sales.IntegrationTests/Contract/ProblemDetailsContractTests.cs:15) and, since 2026-08-13, all four ADC services (MMCA.ADC/Tests/Integration/MMCA.ADC.Conference.IntegrationTests/Contract/ProblemDetailsContractTests.cs:19, MMCA.ADC.Engagement.IntegrationTests/Contract/ProblemDetailsContractTests.cs:16, MMCA.ADC.Identity.IntegrationTests/Contract/ProblemDetailsContractTests.cs:16, MMCA.ADC.Notification.IntegrationTests/Contract/ProblemDetailsContractTests.cs:15): ADC Notification gained the missing subclass, merged to main on 2026-08-13, so every REST host in both consumers is now guarded for this one contract. The versioning contract is subclassed once per repo, on ADC Conference (ApiVersioningTests.cs:14) and Store Catalog (MMCA.Store/Tests/Integration/MMCA.Store.Catalog.IntegrationTests/Contract/ApiVersioningTests.cs:15), which is enough to keep the machinery exercised but leaves the other five REST hosts unguarded. The security-headers suite is subclassed on all four browser-facing hosts: the two Gateway hosts (MMCA.Store/Tests/Hosts/MMCA.Store.Gateway.Tests/SecurityHeadersTests.cs:12, MMCA.ADC/Tests/Hosts/MMCA.ADC.Gateway.Tests/SecurityHeadersTests.cs:12) and, since 2026-09-10, both UI web hosts (MMCA.Store/Tests/Hosts/MMCA.Store.UI.Web.Tests/SecurityHeadersTests.cs:17-18, MMCA.ADC/Tests/Hosts/MMCA.ADC.UI.Web.Tests/SecurityHeadersTests.cs:17-18, each over a Production-pinned factory: StorefrontHostApplicationFactory and MMCA.ADC/Tests/Hosts/MMCA.ADC.UI.Web.Tests/ConferenceUiHostApplicationFactory.cs:17). Graceful shutdown remains Gateway-only (the two GracefulShutdownTests above); no service host asserts it today. The gateway-hardening suite is subclassed on both Gateway hosts and nowhere else, which is its whole addressable surface: it asserts the shared gateway kit's edge behavior, and only a gateway adopts that kit (MMCA.ADC/Tests/Hosts/MMCA.ADC.Gateway.Tests/GatewayHardeningTests.cs:30, MMCA.Store/Tests/Hosts/MMCA.Store.Gateway.Tests/GatewayHardeningTests.cs:35). Each subclass states only its own route-table facts and supplies a Production-pinned factory with a recording forwarder standing in for IHttpForwarder (MMCA.ADC/Tests/Hosts/MMCA.ADC.Gateway.Tests/GatewayHardeningTests.cs:103, MMCA.Store/Tests/Hosts/MMCA.Store.Gateway.Tests/GatewayHardeningTests.cs:117). The decorator suite is subclassed once per consumer repo and is the one base all three of them run: ADC and Store both against the Identity module's ChangePreferencesCommand / GetUserPreferencesQuery pair (MMCA.ADC/Tests/Architecture/MMCA.ADC.Architecture.Tests/DecoratorPipelineOrderTests.cs:27, MMCA.Store/Tests/Architecture/MMCA.Store.Architecture.Tests/DecoratorPipelineOrderTests.cs:26), and MMCA.Helpdesk against a real Tickets pair, described below. MMCA.Common dogfoods the only base it can, since it ships no host of its own: a synthetic PingCommand/PingQuery pair driven through the framework's own registration sequence (Tests/Hosting/MMCA.Common.Testing.Tests/Conformance/DecoratorPipelineOrderTests.cs:23).

MMCA.Helpdesk adopts exactly one of them, the decorator suite, having adopted none of them when this record was written. Its architecture-test project now references MMCA.Common.Testing alongside the structural package (MMCA.Helpdesk/Tests/Architecture/MMCA.Helpdesk.Architecture.Tests/MMCA.Helpdesk.Architecture.Tests.csproj:26) and subclasses DecoratorPipelineOrderTestsBase against a real Tickets pair: the framework's generic UpdateEntityCommand<Ticket, TicketUpdateRequest, TicketIdentifierType>, closed over the Ticket aggregate by the seed's own AddEntityCrud wiring, and the module's hand-written GetTicketByIdQuery (MMCA.Helpdesk/Tests/Architecture/MMCA.Helpdesk.Architecture.Tests/DecoratorPipelineOrderTests.cs:37, :39), whose ConfigureServices runs the seed's own registration sequence: AddApplication(), the Tickets module registration, then AddApplicationDecorators() last (:59-61). Two files in that project import the MMCA.Common.Testing.Conformance namespace, and only one of them belongs to this record: that decorator subclass (:13). The other, MiddlewarePipelineOrderTests.cs (:1), is a one-line subclass of MiddlewarePipelineOrderTestsBase (MMCA.Common/Source/Hosting/MMCA.Common.Testing/Conformance/MiddlewarePipelineOrderTestsBase.cs:29, MMCA.Helpdesk/Tests/Architecture/MMCA.Helpdesk.Architecture.Tests/MiddlewarePipelineOrderTests.cs:15) guarding the ADR-079 shared HTTP edge-pipeline order, not one of the seven contract bases above. The package reference the domain-test project carries (MMCA.Helpdesk/Tests/Modules/Tickets/MMCA.Helpdesk.Tickets.Domain.Tests/MMCA.Helpdesk.Tickets.Domain.Tests.csproj:8) is still unused. The six booted-host bases have no Helpdesk subclass, so the seed's three test projects remain the Tickets domain and application suites plus the architecture suite, which now carries both tiers: the ADR-015 structural bases from the separate MMCA.Common.Testing.Architecture package (MMCA.Helpdesk/Tests/Architecture/MMCA.Helpdesk.Architecture.Tests/GlobalUsings.cs:3), and from MMCA.Common.Testing one base from this record's set plus the ADR-079 edge-pipeline base.

Rationale

  • Runtime conformance is the half ADR-015 excluded. Structural rules answer "is the code shaped correctly"; these suites answer "does the composed host behave correctly". A host that registers everything the framework asks for can still drop a security header in a middleware reorder, wedge a rolling deploy with a hosted service that will not drain, or silently invert the decorator pipeline. None of that is reachable by reflection over assemblies.
  • The failure this catches does not announce itself in production. A non-draining host looks healthy until a deploy waits out its termination grace period; a missing HSTS header looks healthy until an audit; a reordered decorator pipeline looks healthy until a validation runs inside a transaction that should never have opened.
  • Write the body once, subclass thin. Same lever as ADR-015: the assertions live in the package, so a new invariant reaches every consumer that already subclassed the base. The gateway shutdown subclasses are single lines with no body, which is the intended cost of adoption.
  • Booting the host is what makes it honest. Pinning Production exercises the branches a default Development boot skips, and a throwaway migrated database means the schema under test is the one the host's own init strategy produced, not a fixture's guess.
  • A live document beats a snapshot. Asserting against the served OpenAPI document removes the class of failure where the guard passes because the snapshot was regenerated along with the regression.

Trade-offs

  • Opt-in per host, exactly like ADR-015. The framework ships the suites; a host gets the gate only once someone writes the subclass. That is the same audit-the-inventory caveat, and the adoption inventory above is the current answer to it, not a claim of completeness.
  • Coverage is uneven by suite. Graceful shutdown is Gateway-only, versioning is one host per repo, and Helpdesk has only the decorator suite. Problem details is the one suite subclassed on every REST host in both consumers (ADC Notification closed the last gap on 2026-08-13). Security headers is complete over the hosts it addresses rather than over all of them: both Gateway hosts and both UI web hosts subclass it, which is every host that serves a browser, and the four extracted REST services are reached through a gateway that stamps the headers for them. Gateway hardening is Gateway-only by construction rather than by omission: it asserts the shared gateway kit, so a service host has nothing for it to check. Every remaining hole is an unguarded host for that contract, not a decision that the contract does not apply.
  • Some of the suites need a real SQL Server. The three fixture-driven suites (problem details, OpenAPI, versioning) live in the per-service integration tier and cannot run in a database-free test pass; the three Gateway suites (security headers, graceful shutdown, gateway hardening: no DbContext, no broker) and the decorator suite run in the fast tier. That splits the runtime gate across two CI jobs with different prerequisites.
  • The assertions are coarse by construction. MinimumPathCount is a floor, CorePublicResources checks presence rather than exact casing or schema (OpenApiContractTestsBase.cs:80), and the problem-details base checks shape (status, title, a diagnostic extension) rather than message content. These catch wholesale regressions, not subtle ones.
  • The decorator check reads private fields. Unwrapping the chain depends on each decorator holding its inner handler in a field that implements the same closed interface (DecoratorPipelineOrderTestsBase.cs:112-115); a decorator that stored it differently would silently end the walk early rather than fail loudly.
  • A new contract base only reaches consumers at the next lockstep bump. The suites ship inside the package set, so adding one is a framework release plus a consumer sweep (ADR-016), not a local edit in the repo that needs the guard.

Revision (2026-09-10)

The security-headers suite reaches both UI web hosts, so the adoption inventory above is no longer Gateway-only for it. ADC added MMCA.ADC/Tests/Hosts/MMCA.ADC.UI.Web.Tests, whose SecurityHeadersTests.cs:17-18 is the one-line subclass and whose ConferenceUiHostApplicationFactory.cs:17 derives the framework's ProductionHostApplicationFactory<Program> so the host boots on the environment that emits HSTS. The project is listed in MMCA.ADC/MMCA.ADC.CI.slnf:61, the no-database tier, matching where Store's equivalent sits (MMCA.Store/MMCA.Store.CI.slnf:55): adoption here means running on every pull request, not merely existing.

That project also carries two ADC-local hardening tests over the same factory, which are not framework-suite subclasses and are counted as neither: UiRateLimitingTests.cs:16 (burst shedding at the configured window, exempt paths, per-IP partitioning, and the shipped values) and BoundedCircuitHandlerTests.cs:16 (the active-circuit ceiling, permit release, and the retention limits). They belong to ADR-088's own-host hardening rather than to this record, and they are noted here only so the inventory above is not read as the whole content of that test project.

ADR-015 (the structural / registration fitness layer this complements; its stated non-goal, "not runtime behavior", is exactly this ADR's scope, and the two tiers ship as two separate packages), ADR-014 (the decorator execution order DecoratorPipelineOrderTestsBase proves at runtime), ADR-013 (the RFC 9457 ProblemDetails edge contract ProblemDetailsContractTestsBase guards), ADR-046 (HTTP API versioning, whose "shared fitness contract" is one of these bases), ADR-079 (the shared HTTP edge pipeline, whose own order-checking base ships beside these in the same Conformance folder but is not one of these seven), ADR-016 (lockstep versioning: a new conformance base reaches consumers only through a release and a full sweep). Package inventory for the framework lives in MMCA.Common/FACTS.md.

← Back to Architecture Decision Records View source on GitHub ↗