Ivan Ball-llovera

↑↓ to navigate Enter to open "…" all these words ANDOR to combine

Architecture Decision Record

ADR-041: Observability and Telemetry Strategy

Status

Accepted (2026-07-10). Amended (2026-07-23) to document the Telemetry:DisableHttpClientMetrics and Telemetry:DisableRuntimeMetrics cost knobs and to correct the meter/activity-source literal citations. Amended (2026-07-25) to describe how the CQRS logging decorators actually record duration (a per-path RecordDuration helper, not a finally) and to rebase the decorator outcome-tag, outbox poll-span, and OutboxProcess parent-context citations onto their current lines. Amended (2026-07-28) to rebase the Aspire service-defaults citations (the sampling and cost-knob helpers, the two exporter reads) and the outbox dead-letter increment onto their current lines. Amended (2026-08-01) to rebase the ASP.NET Core/HttpClient tracing citation, the CQRS duration-literal citations, the outbox poll-span-filter clear statement, the outbox meter/counter/activity-source declarations, the dead-letter increment call site, the poll-span open site, the OutboxProcess span-start site, and the correlation-id response-header citation onto their current lines. Amended (2026-08-07) to record that the outbox meter and dead-letter counter now live in a dedicated OutboxMetrics type rather than in OutboxProcessor, to document the second dead-letter increment on the retries-exhausted path and the reason tag that separates the two, and to rebase the Aspire service-defaults, CQRS metric-literal, and outbox span citations onto their current lines. Amended (2026-08-18) to record two new meter families (MMCA.Common.OutputCache, MMCA.Common.BestEffort), to correct the meter inventory this record has been under-reporting, and to note that the correlation id now starts one hop earlier, at the Gateway; see the Revision (2026-08-18) at the end. Amended (2026-08-31) to record the application logging pipeline (Serilog as one additional provider beside the OpenTelemetry one, never a replacement ILoggerFactory), its level and file-sink policy, the pre-DI bootstrap logger, and which hosts adopt it; see the Amended (2026-08-31) section at the end. Amended (2026-09-03): MMCA.Store.UI.Web now calls AddCommonSerilog instead of hand-rolling the same configuration inline, so the "defaults exist in two shapes" cost that amendment recorded is gone and eight hosts share one helper; its Program.cs citations and the per-host AddCommonSerilog / bootstrap-factory line anchors are rebased onto their current lines. Amended again (2026-09-03) to record the probe-telemetry cost knob (Telemetry:FilterProbeTelemetry, the one knob that is on by default) and the metric-drop views that make the two metrics knobs authoritative over the Azure Monitor distro, and to rebase the Aspire service-defaults, CQRS decorator and outbox citations (the outbox processor and its metrics now live under Persistence/Outbox/Processing/) onto their current lines; see the "Amended (2026-09-03): probe telemetry" section at the end.

Context

The framework is a modular monolith whose modules extract into standalone services (ADR-008), so the same telemetry has to make sense whether a request stays in one process or crosses a gateway and several service hosts. OpenTelemetry auto-instrumentation (ASP.NET Core, HttpClient, the .NET runtime) gives generic HTTP and runtime signals for free, but it is blind to the two paths that carry almost all of the framework's own work: the CQRS use-case pipeline (ADR-014) and the outbox (ADR-003). "How long is this command taking and how often does it fail" and "is the outbox dead-lettering" are not questions auto-instrumentation can answer.

Two cost forces pull the other way. A deployed fleet polls every relational outbox around the clock, so idle poll spans would dominate Application Insights ingestion if exported, and full-fidelity tracing is the single largest observability line item. The framework needs custom instrumentation where auto-instrumentation is blind, plus knobs that cut telemetry cost without going dark. This cross-cutting observability decision was implemented but named by no existing ADR; this record captures it.

Decision

Standardize telemetry in the shared Aspire service defaults, add framework-specific instrumentation for the CQRS and outbox paths, and expose cost knobs with fail-safe defaults.

  • One shared telemetry baseline on every host. ConfigureOpenTelemetry (Source/Hosting/MMCA.Common.Aspire/Extensions.cs:121) wires OpenTelemetry logging with formatted messages and scopes (Extensions.cs:132-Extensions.cs:133), metrics from ASP.NET Core (unconditional, Extensions.cs:139) plus HttpClient and the runtime (each gated behind a cost knob, see below), and tracing from ASP.NET Core and HttpClient, added either with the probe-telemetry filters attached (Extensions.cs:230-Extensions.cs:233) or plain (Extensions.cs:237-Extensions.cs:238) depending on the knob the Amended (2026-09-03) section records. It is called from AddServiceDefaults (Extensions.cs:41), so a host opts in once and every project in the Aspire model inherits the same pipeline.

  • Custom RED metrics from the CQRS pipeline. A single meter MMCA.Common.Cqrs (Source/Core/MMCA.Common.Application/UseCases/Decorators/CqrsMetrics.cs:24) publishes two duration histograms: cqrs.command.duration (CqrsMetrics.cs:30) and cqrs.query.duration (CqrsMetrics.cs:36), both in milliseconds. Every path is measured without a finally: each logging decorator routes all three of its exits through a private RecordDuration helper, so the measurement cannot be skipped. The command helper calls CqrsMetrics.CommandDuration.Record(...) tagged by command and outcome (Source/Core/MMCA.Common.Application/UseCases/Decorators/LoggingCommandDecorator.cs:80, in the RecordDuration helper declared at :79) and the query helper does the same for QueryDuration (Source/Core/MMCA.Common.Application/UseCases/Decorators/LoggingQueryDecorator.cs:78, helper at :77). The outcome tag takes completed, failed (a Result failure), or exception, one call site per path (LoggingCommandDecorator.cs:49, :44, :58; the query equivalents at LoggingQueryDecorator.cs:46, :41, :55), so count gives rate, the tag gives errors, and the histogram gives duration. The Aspire host subscribes the meter by literal name (Extensions.cs:200).

  • An outbox dead-letter counter. The outbox instruments live in their own static type (Source/Core/MMCA.Common.Infrastructure/Persistence/Outbox/Processing/OutboxMetrics.cs:16), which owns the meter MMCA.Common.Outbox (OutboxMetrics.cs:19) and the counter outbox.dead_letter.count (OutboxMetrics.cs:41-OutboxMetrics.cs:42). OutboxProcessor (Source/Core/MMCA.Common.Infrastructure/Persistence/Outbox/Processing/OutboxProcessor.cs) increments it on both dead-letter paths, tagged by event_type and by a reason that tells them apart: type_unresolvable when a message's event type cannot be resolved (OutboxProcessor.cs:719-OutboxProcessor.cs:722), and retries_exhausted when a failing message reaches MaxRetries and drops out of the poll (OutboxProcessor.cs:674-OutboxProcessor.cs:677). The processor's activity source publishes outbox spans under the same name (OutboxProcessor.cs:87); both the meter and the trace source are registered by literal name in the Aspire defaults (Extensions.cs:199, Extensions.cs:210).

  • Correlation-ID middleware ties the request together. CorrelationIdMiddleware (Source/Presentation/MMCA.Common.API/Middleware/CorrelationIdMiddleware.cs:15) uses the X-Correlation-ID header (CorrelationIdMiddleware.cs:18), reading it from the request or falling back to the current W3C trace id and then to HttpContext.TraceIdentifier (CorrelationIdMiddleware.cs:32), sets it on the scoped ICorrelationContext (CorrelationIdMiddleware.cs:36), and echoes it on the response (CorrelationIdMiddleware.cs:39, inside the OnStarting callback registered at CorrelationIdMiddleware.cs:37). The CQRS logging decorators stamp that same id into every log scope (read at LoggingCommandDecorator.cs:25, stamped by BeginCommandScope at :27), so logs, the correlation id, and the trace id line up for one request.

  • Two high-volume metric families gated behind cost knobs, on by default. ASP.NET Core metrics are always wired (Extensions.cs:139), but the two heaviest AppMetrics contributors on a low-traffic multi-service deployment are conditional. HttpClient connection and request metrics are added only when Telemetry:DisableHttpClientMetrics is unset or false (Extensions.cs:148, adding instrumentation at Extensions.cs:168), and .NET runtime metrics (dotnet.gc.*, jit.*, thread_pool.*) only when Telemetry:DisableRuntimeMetrics is unset or false (Extensions.cs:175, adding at Extensions.cs:187). Skipping the instrumentation is not enough on its own, so each disabled branch also drops the whole meter with a View (Extensions.cs:160-Extensions.cs:164 for System.Net.Http plus System.Net.NameResolution, Extensions.cs:180-Extensions.cs:183 for System.Runtime): the Azure Monitor distro adds those meters itself, and a View applies to the whole MeterProvider regardless of which component added them, which is what makes each knob authoritative rather than advisory. Both keys are read by IsInstrumentationDisabled (Extensions.cs:471-Extensions.cs:472), which drops the family only when the value parses as boolean true; absent, blank, or unparseable falls back to keeping the instrumentation, so a typo cannot silently blind a whole metric family. A deployed host sets one or both to true to cut ingestion cost; outbound dependency latency is still captured as traces when HttpClient metrics are dropped.

  • Head-based sampling as a cost knob, off by default. Telemetry:TracesSampleRatio (Extensions.cs:261, parsed by TryGetTraceSampleRatio at Extensions.cs:448, which reads the key at Extensions.cs:451) is unset by default, so a host samples everything and behavior does not change. A deployed host sets a ratio in the open interval (0,1) to keep that fraction of traces; the value wraps a TraceIdRatioBasedSampler in a ParentBasedSampler (Extensions.cs:262) so a sampled-in request keeps its whole trace across service boundaries. A key that is absent, unparseable, or outside (0,1) falls back to sample-all (Extensions.cs:452-Extensions.cs:457), so a typo can never silently drop all telemetry.

  • Outbox poll spans are filtered out of export. OutboxPollFilterProcessor (Source/Hosting/MMCA.Common.Aspire/Telemetry/OutboxPollFilterProcessor.cs:15), registered before the exporters (Extensions.cs:246), clears the Recorded flag on the recurring OutboxPoll span and its children (OutboxPollFilterProcessor.cs:45). The poll query runs inside that span, opened at the top of FetchCandidatesAsync (OutboxProcessor.cs:413, span started at OutboxProcessor.cs:419, named at OutboxProcessor.cs:75), so steady-state polling does not flood Application Insights. Real outbox work is untouched: each per-message OutboxProcess span is started by StartOutboxActivity (called once per message at OutboxProcessor.cs:579, declared at OutboxProcessor.cs:775) under an explicit parent context restored from the message's stored trace and span ids (OutboxProcessor.cs:782-OutboxProcessor.cs:785), span started at OutboxProcessor.cs:787-OutboxProcessor.cs:790, so it is never a child of the poll span.

  • Dual exporters, either or both. AddOpenTelemetryExporters enables OTLP when OTEL_EXPORTER_OTLP_ENDPOINT is present (Extensions.cs:363-Extensions.cs:364, the Aspire dashboard sets it, exporter wired at Extensions.cs:368) and Azure Monitor via UseAzureMonitor (Extensions.cs:376) when APPLICATIONINSIGHTS_CONNECTION_STRING is present (read at Extensions.cs:371-Extensions.cs:372, checked at Extensions.cs:374, and set by the cloud deployment). Both can be active at once (Extensions.cs:359), so local development ships to the Aspire dashboard and production ships to workspace-based Application Insights with no code change.

Rationale

  • Instrument only where auto-instrumentation is blind. The CQRS RED histograms and the outbox dead-letter counter cover the two framework-owned hot paths; everything else (HTTP, runtime) rides the free auto-instrumentation, so the custom surface stays small.
  • RED at the decorator, not in every handler. The CQRS pipeline already wraps every handler in a logging decorator (ADR-014), so recording duration and outcome there makes metrics a byproduct of a pipeline layer that exists, with no per-handler discipline (the invariant-over-discipline posture, ADR-015).
  • A single correlation id with a W3C fallback. Whether or not a client supplies X-Correlation-ID, one id stitches the logs of a request together and matches the trace, which is what an operator needs first when a distributed call goes wrong.
  • Cost knobs default to safe. Sampling, poll-span filtering, and the HttpClient/runtime metric toggles are the levers a FinOps owner reaches for (COST.md), and all fail toward keeping data: sampling is off unless configured, an out-of-range ratio is ignored, only idle poll spans are dropped, and a metric family drops only on an explicit boolean true (a typo keeps it on).
  • ParentBased keeps distributed traces coherent. An extracted-service deployment (ADR-008) needs a sampled-in request to stay sampled end to end; a per-hop random sampler would shred cross-service traces.

Trade-offs

  • Custom instrumentation carries a maintenance cost. The Aspire package has no reference to Application or Infrastructure by design, so the meter and activity-source names are duplicated as literals (the meter subscriptions at Extensions.cs:199-Extensions.cs:205 and the trace source at Extensions.cs:210, and the sync notes at CqrsMetrics.cs:8, OutboxMetrics.cs:8 and OutboxPollFilterProcessor.cs:17). A rename on one side silently stops export until the literal is updated. That is the price of the decoupled package graph.
  • Sampling trades trace completeness for cost. A sampled-out trace is simply gone; deep debugging of a specific request can miss it. Metrics and logs are unaffected (sampling is trace-only), so RED rates and error counts stay whole even at a low ratio.
  • Poll-span filtering hides steady-state outbox activity. The dead-letter counter and per-message OutboxProcess spans remain, but "is the poller alive and looping" cannot be answered from traces alone, by design (that signal is metrics and the dead-letter counter, not spans).
  • Cross-service trace continuity depends on stored ids and the parent decision. A linked OutboxProcess trace only reconnects when the producer captured the trace and span ids on the message; ParentBased sampling that dropped the originating trace also drops the linked span.
  • Exporters and sampling are opt-in per host. A host that sets neither exporter variable emits to nothing, and a misconfigured ratio fails toward sample-all (higher cost) rather than toward silence: the intended bias, but it means a cost surprise is possible where a data gap is not.

Revision (2026-08-18)

Two meters and one hop.

Two new failure counters, each on its own meter. cache.eviction.failed, tagged cache_tag, on MMCA.Common.OutputCache (MMCA.Common/Source/Presentation/MMCA.Common.API/Caching/OutputCacheMetrics.cs:19, instrument at :29-37) counts a cross-service output-cache eviction that failed for one tag (ADR-026's Revision (2026-08-18)); and besteffort.dispatch.failed, tagged operation, on MMCA.Common.BestEffort (MMCA.Common/Source/Core/MMCA.Common.Application/Services/BestEffort.cs:102, instrument at :107-115) counts a swallowed fire-and-forget side effect, the helper's whole purpose being that the caller does not see the failure. Both are subscribed in the Aspire defaults (MMCA.Common/Source/Hosting/MMCA.Common.Aspire/Extensions.cs:204-205).

The meter inventory in the Decision above is wrong and has been for a while. This record names two meters, and ADR-087 called MMCA.Common.Broker "a third", which was already an undercount. The authoritative list is the subscription block itself (Extensions.cs:199-205), which now carries seven: MMCA.Common.Outbox, MMCA.Common.Cqrs, MMCA.Common.Idempotency, MMCA.Common.Scheduler, MMCA.Common.Broker, MMCA.Common.OutputCache, MMCA.Common.BestEffort. Two of them (Idempotency, Scheduler) were never recorded here at all. Read that block, not this prose, when the question is what the framework exports.

Correlation now starts at the edge. ADR-088 adds a context-free GatewayCorrelationMiddleware that ensures X-Correlation-ID on the way in and echoes it on the way out, writing it onto the forwarded request so the service-tier CorrelationIdMiddleware adopts it rather than minting its own. The Decision's claim that one id stitches a request together becomes true across the gateway hop, where it previously began at the first service and left the Gateway's own logs unlinked. No meter, no span, one header, one hop earlier.

Two costs come with it. Both new counters are failure-only, so a healthy system emits nothing on them and a zero is indistinguishable from a host that never wired the feature, which is exactly the shape of signal that goes unnoticed until an incident. And neither is wired to an alert or a runbook section, joining ADR-087's two counters in the gap ADR-062 describes. The duplicated-literal cost this record already records in Trade-offs now applies to seven names rather than two.

Amended (2026-08-31)

The log side of this record. Until now it named only the OpenTelemetry logging call inside ConfigureOpenTelemetry (Extensions.cs:130); what a host actually WRITES its application log lines through was undocumented.

Serilog is registered as ONE additional provider, never through UseSerilog(). AddCommonSerilog (MMCA.Common/Source/Hosting/MMCA.Common.Aspire/Logging/SerilogHostExtensions.cs:48) builds the framework's logger configuration, publishes it as the global Log.Logger (SerilogHostExtensions.cs:54), and adds it to the host's existing factory with builder.Logging.AddSerilog(Log.Logger, dispose: true) (:55). The alternative is a silent-failure trap no code reading surfaces, which is why the rationale lives on the type itself (:16-:20): UseSerilog() replaces the whole ILoggerFactory and with it every other provider, including the OpenTelemetry to Azure Monitor provider AddServiceDefaults wires (Extensions.cs:41, Extensions.cs:121). A host that calls it publishes no application log line to Application Insights at all, while its metrics, traces and health endpoints stay green, so the gap reads as a quiet service rather than as a misconfiguration. Ordering carries the same weight in the other direction: the helper runs BEFORE AddServiceDefaults() in every host that uses it (for example MMCA.ADC/Source/Services/MMCA.ADC.Conference.Service/Program.cs:100-:101), so the OpenTelemetry provider joins the factory Serilog is already in. One fitness test pins the invariant: the built container must contain exactly one SerilogLoggerProvider (MMCA.Common/Tests/Hosting/MMCA.Common.Aspire.Tests/Logging/SerilogHostExtensionsTests.cs:156-:158).

Level policy and sinks. The minimum level is Debug in Development and Information everywhere else (ResolveMinimumLevel, SerilogHostExtensions.cs:76-:77, applied at :107), with Microsoft.EntityFrameworkCore and Microsoft.AspNetCore held at Warning (:108-:109) and a console sink always (:110). The rolling daily file sink is environment-conditional: added everywhere except Production (ShouldWriteFileSink, :87-:88, applied at :112-:118), because a production container writes it to ephemeral disk nothing reads while stdout and the OpenTelemetry provider already carry the same events, whereas outside Production (local runs and the CI E2E stack) that file is what a failure gets diagnosed from. A host needing one extra sink or enricher passes the optional configure hook (:50, invoked at :120) instead of forking the helper.

A bootstrap logger for the pre-DI window. Module discovery runs before the DI container exists, so there is no ILogger<T> to resolve yet. CreateBootstrapLoggerFactory() (:67-:68) returns a factory writing to the same global Log.Logger, which each host disposes once startup wiring is done (MMCA.Store/Source/Services/MMCA.Store.Catalog.Service/Program.cs:122).

Adoption is asymmetric, but there is only one shape of it. Eight hosts call AddCommonSerilog, each in its own Program.cs: the seven ADC/Store service hosts, ADC Conference (:100), Engagement (:83), Identity (:96), Notification (:86), and Store Catalog (:68), Identity (:73), Sales (:85), plus MMCA.Store.UI.Web (MMCA.Store/Source/Hosts/UI/MMCA.Store.UI.Web/Program.cs:58). Each of the seven service hosts pairs it with CreateBootstrapLoggerFactory() in the same file (ADC :308, :209, :247, :187; Store :122, :117, :131); MMCA.Store.UI.Web does not, because it discovers no modules and so has no pre-DI window to cover. The asymmetry that remains is the three hosts with no Serilog at all: neither Gateway references it in its Program.cs, and neither does MMCA.ADC.UI.Web. Those three take the plain OpenTelemetry logging AddServiceDefaults gives them.

Two costs come with it. The invariant is guarded in the framework, not at the consumer: the one test above runs in MMCA.Common, and nothing in a host's own build stops a new service from reaching for UseSerilog(), whose failure mode is silence in a place nobody watches for absence. And Information is not the level that reaches the queryable store: ADR-098 records the production thinning that floors the OpenTelemetry logging provider at Warning while Serilog keeps Information on container stdout, so the two providers this record puts side by side deliberately carry different volumes.

Amended (2026-09-03): probe telemetry

Health probes were the trace bill. Container Apps liveness and readiness probes, the gateway's downstream aggregate probes, YARP active health checks and the availability web test accounted for every AppRequests row in both production workspaces, and their children (the health check's SQL SELECT 1, the Redis PING, the gateway's HttpClient calls to each backend's /alive) for most of the AppDependencies rows. None of it carries end-user signal, and none of it is touched by Telemetry:TracesSampleRatio, because probe spans are exactly what a ratio sampler is asked to keep proportionally.

A third cost knob, and the only one that defaults to on. Telemetry:FilterProbeTelemetry (Extensions.cs:35) is read by IsProbeTelemetryFilterEnabled (Extensions.cs:483-Extensions.cs:484) at Extensions.cs:224. It inverts the fail-safe direction of the other knobs on purpose: absent, blank or unparseable all mean "filter", and only an explicit boolean false turns filtering off, for a host debugging its own probes. What a probe path is comes from one place, HealthEndpointPaths.IsProbePath (Source/Hosting/MMCA.Common.Aspire/HealthEndpointPaths.cs:29-:33): /alive, /health, and anything below /health/, case-insensitively.

Two instrumentation predicates plus one processor, because probe spans arrive by three routes. With the knob on, the tracing setup attaches both filters to the default-named instrumentation options (Extensions.cs:230-Extensions.cs:233; the unfiltered branch at Extensions.cs:237-Extensions.cs:238 is plain AddAspNetCoreInstrumentation and AddHttpClientInstrumentation). ProbeTelemetryFilter.ShouldCollectRequest (Source/Hosting/MMCA.Common.Aspire/Telemetry/ProbeTelemetryFilter.cs:40) refuses the inbound probe request span and stamps an mmca.probe marker tag on it (ProbeTelemetryFilter.cs:33, set at :51), because a refused request never gets its url.path written and its descendants would otherwise have no way to recognize their own ancestor. ProbeTelemetryFilter.ShouldCollectOutgoing (:62-:63) refuses outbound probe calls that are not descendants of any inbound request, the gateway's DownstreamServiceHealthCheck calls and YARP's active checks, both driven by background timers. The descendants are handled by ProbeTelemetryFilterProcessor (Source/Hosting/MMCA.Common.Aspire/Telemetry/ProbeTelemetryFilterProcessor.cs:20), registered only when the knob is on and, like the outbox poll filter, before the exporters (Extensions.cs:253): it walks the in-process parent chain (ProbeTelemetryFilterProcessor.cs:52), matches the marker or a server span whose path, route or display name is a probe (:66-:79), and clears Recorded plus IsAllDataRequested (:59-:60) at both OnStart (:29) and OnEnd (:40), since a client span carries no identifying tag yet when it starts. Unlike the two metrics knobs, these filters need no view: configuring the default-named options also covers the instrumentation the Azure Monitor distro adds.

Metrics are deliberately untouched (Extensions.cs:222-Extensions.cs:223): http.server.request.duration, Kestrel and routing instruments keep flowing, so probe traffic stays on dashboards.

Two costs come with it. "Did the probe pass" is no longer answerable from traces, the same blindness poll-span filtering already accepts for the outbox, so that question belongs to metrics and the health endpoints (ADR-025) instead. And this knob fails toward dropping data while sampling and the two metrics toggles fail toward keeping it: a host that adds a real route below /health/ has its traces filtered by IsProbePath's prefix match with no error and no log line.

ADR-003 (the outbox whose dead-letter counter and poll-span filtering this defines), ADR-014 (the CQRS decorator pipeline that emits the RED histograms as a byproduct of its logging decorators), ADR-009 (resilience and recovery objectives, configured alongside telemetry in the same AddServiceDefaults; observability is the diagnostic layer under that posture), ADR-025 (startup warm-up and readiness gating, whose health-check endpoints are the operational-signal sibling of these telemetry signals in the same Aspire defaults), and COST.md (the FinOps companion that records span-filtering and sampling as cost levers).

← Back to Architecture Decision Records View source on GitHub ↗