Ivan Ball-llovera

↑↓ to navigate Enter to open "…" all these words ANDOR to combine

Architecture Decision Record

ADR-052: Background Job Execution (Bounded Queue plus Hosted Drain)

Status

Accepted (2026-07-24). Revised 2026-08-23 (post-commit enqueue is recorded as two patterns, not one) and 2026-08-31 (three of the four command-handler enqueue sites inherit the save-then-enqueue ordering from a shared base class instead of writing it): see the revisions at the end.

Context

Some requests trigger work that cannot run inside the request: an AI scoring pass over an event's sessions takes minutes and issues one paid API call per session, and a live-channel broadcast must not put a gRPC round trip on the hot path of a vote. The endpoint should accept the request and return, leaving the work to run elsewhere.

The obvious shortcut is to start an untracked task and return, which is what the session-scoring endpoint did (_ = RunScoringInBackgroundAsync(eventId)). A code review found three defects that follow from the shape rather than from that particular implementation:

  • The host does not know the work exists. IHostApplicationLifetime cannot wait for it and shutdown cannot cancel it, so an Azure Container Apps scale-in or a deploy tears it down mid-run with nothing recorded. For a multi-minute paid pass, that is spend with no result and no trace.
  • Nothing deduplicates it. Two clicks started two concurrent passes over the same event: double the API calls, racing each other's writes, and no way for the caller to learn a run was already in flight.
  • Failure handling drifts. Each fire-and-forget site invents its own try/catch, so the failure posture (swallow, log, retry) is decided per call site instead of once.

There is a second, unrelated need with the same shape: work that must happen after the request's transaction commits and must not add latency to it. The live-channel broadcast is that case, and it already used a bounded queue plus a hosted drain (ADR-039). The two wanted the same mechanism.

This is deliberately not a distributed job system. Every case here is best-effort or re-triggerable work owned by one service; durable, cross-process scheduling is what the outbox (ADR-003) and the broker (ADR-008) are for, and this ADR does not compete with them.

Decision

In-process background work runs as a bounded queue plus a single-reader hosted drain. Nothing starts an untracked Task from a request.

  • A bounded Channel<T> per job kind, registered as a singleton, with the concrete type and its interface both resolving to the one instance (TryAddSingleton<TQueue>() plus TryAddSingleton<IQueue>(sp => sp.GetRequiredService<TQueue>())). Registering them separately would give producers a queue nobody drains.
  • A BackgroundService drain per queue, SingleReader, consuming with ReadAllAsync(stoppingToken). Because it is a hosted service the host owns the work: shutdown cancels it and waits for it to unwind. The drain resolves scoped services through IServiceScopeFactory per item, since it is itself a singleton.
  • The full mode encodes what the work is worth.
    • Ephemeral work uses BoundedChannelFullMode.DropOldest: under backpressure the freshest broadcast is worth more than the oldest and the request path must never block. Note that TryWrite then always returns true (it evicts to make room), so a caller cannot learn from its return value that anything was dropped; the channel's itemDropped callback is the only real signal and must be wired to a counter and a log.
    • Expensive work uses Wait with a non-blocking TryWrite, so a full queue refuses the request rather than silently discarding an earlier one, and the caller is told.
  • Expensive work deduplicates by its natural key, with the claim taken before the write and released by the drain only when the run finishes. The dedup window therefore covers execution, not just the wait in the queue. A duplicate request gets an explicit refusal (409), not a silent coalesce.
  • One failure posture per drain, stated once. The drain catches per item so one failed run cannot kill the loop, and handles shutdown cancellation separately from failure so a graceful restart is not recorded as an error.
  • Post-commit work is enqueued after the write is durable, never beside it. The failure this rules out is enqueuing while the write can still be undone, which leaves the queued work describing state that never persisted. Two shapes satisfy it and both are in use:
    • From a domain event handler, which gets post-commit delivery from the existing deferral (ADR-003) with no sequencing code in the command handler: domain-event dispatch inside a transactional command is deferred until after the commit succeeds and dropped on rollback.
    • From the command handler side of a non-transactional command, once its save has returned, for commands that do not opt into the ADR-014 transaction decorator (which wraps only commands implementing ITransactional, so for the rest SaveChangesAsync is the commit). Most of these handlers do not write the ordering themselves: a MutateEntityHandlerBase subclass has no save of its own, and the shared base saves and then calls the OnMutatedAsync post-save hook the handler overrides to enqueue, so the order is structural. A handler that saves inline instead keeps the ordering as statement order, where the enqueue call has to stay below the save.

Two implementations exist: LiveChannelPublishQueue / LiveChannelPublishProcessor (ephemeral, DropOldest, ADR-039) and SessionScoringQueue / SessionScoringProcessor (expensive, Wait, dedup by event id).

Rationale

  • The host lifetime is the point. A BackgroundService is the only in-process shape the host can cancel and await. Everything else in the decision follows from wanting that.
  • The queue is where the policy lives. Capacity, full mode and dedup are properties of the work, and putting them in the queue type means a caller cannot get them wrong: it calls TryEnqueue and reads the outcome.
  • Refusal beats silent coalescing for paid work. An organizer who clicks twice should learn the run is already going, not have the second click vanish into a queue.
  • Single reader gives ordering for free, which the live-channel case needs per session, and serializes expensive runs so two of them cannot contend for the same external rate limit.

Trade-offs

  • In-process only. The queue does not survive a restart and does not span replicas. Accepted because every current job is either ephemeral (a lost broadcast is a missed UI refresh) or re-triggerable (an organizer can click score again). Work that must survive a crash belongs in the outbox (ADR-003), not here.
  • Dedup is per replica. Two replicas can each accept a run for the same event. Today the scoring endpoint is organizer-only and low-traffic, so the exposure is small; making it cluster-wide would need a distributed lock or a claim row, which is the point at which this should become a real job system instead.
  • Backpressure is felt differently by the two modes, and choosing the wrong one is a silent bug in either direction: DropOldest on expensive work discards paid runs, and Wait on ephemeral work turns a slow peer into refused broadcasts. The mode is a per-job decision, not a default.
  • A drain is a serialization point. One reader means a slow item delays the queue behind it. That is wanted for scoring and harmless for broadcasts at the observed conference-day load; a job kind that needs parallelism needs its own queue rather than a wider reader, or ordering is lost.

ADR-003 (the outbox, for work that must be durable and cross-process, and the post-commit deferral this relies on), ADR-008 (the broker, for cross-service work), ADR-014 (the transactional decorator whose commit boundary post-commit work attaches to), ADR-039 (live channel push, the ephemeral instance of this pattern), ADR-025 (startup warm-up, the other hosted-service use in the framework).

Revision (2026-08-23)

The decision is unchanged: post-commit work is still enqueued only once the write is durable. What changed is the record of how. This ADR stated the domain-event handler as the mechanism; the live-channel enqueue call sites split two-to-four the other way, so the decision bullet now names both shapes.

  1. Two call sites enqueue from an IDomainEventHandler. LivePollVoteChangedHandler implements IDomainEventHandler<LivePollVoteChanged> (MMCA.ADC/Source/Modules/Engagement/MMCA.ADC.Engagement.Application/LivePolls/DomainEventHandlers/LivePollVoteChangedHandler.cs:41) and enqueues the poll.results-changed broadcast at :79; SessionQuestionUpvoteChangedHandler implements IDomainEventHandler<SessionQuestionUpvoteChanged> (.../SessionQuestions/DomainEventHandlers/SessionQuestionUpvoteChangedHandler.cs:42) and enqueues the upvote-count broadcast at :80. Both are singletons that open their own scope (:53 and :54 respectively) and wrap the work in BestEffort.ExecuteAsync (:51, :52).
  2. Four call sites enqueue from the command handler, on the line after the save. There is no domain event in between. (Superseded by the Revision (2026-08-31): three of these four handlers hold no save of their own today, and their anchors below have moved. The text is left in place as the record of what was true when written.)
    • CloseLivePollHandler awaits unitOfWork.SaveChangesAsync at .../LivePolls/UseCases/Close/CloseLivePollHandler.cs:70, logs at :72 and calls EnqueueClosed(poll) at :74; the queue write itself is at :95.
    • OpenLivePollHandler saves at .../LivePolls/UseCases/Open/OpenLivePollHandler.cs:85 and calls EnqueueOpened(poll) at :89, queue write at :110.
    • SubmitQuestionHandler saves at .../SessionQuestions/UseCases/Submit/SubmitQuestionHandler.cs:103 and awaits EnqueueSubmittedAsync(question) at :107; that helper is a BestEffort.ExecuteAsync body (:130-131) with one queue write per branch (:142 approved, :157 pending-count).
    • ModerateQuestionHandler saves at .../SessionQuestions/UseCases/Moderate/ModerateQuestionHandler.cs:80 and awaits EnqueueModeratedAsync(...) at :84, queue writes at :138 and :152.
  3. The safety property still holds for those four, because they are not transactional commands. TransactionalCommandDecorator wraps only commands implementing the ITransactional marker and passes everything else straight through (MMCA.Common/Source/Core/MMCA.Common.Application/UseCases/Decorators/TransactionalCommandDecorator.cs:28-29, marker at .../UseCases/Markers/ITransactional.cs:6). No Engagement command implements it, so for these handlers SaveChangesAsync is the commit and an enqueue below it is genuinely post-commit. The difference from shape 1 is where the ordering lives: in the command handler's statement order rather than in the ADR-003 deferral, so it is a rule a future edit to those handlers has to keep rather than one the pipeline keeps for them. (The last sentence is superseded by the Revision (2026-08-31): it describes one of the four handlers, not all four.)

Revision (2026-08-31)

The decision and the safety property are unchanged: post-commit work is still enqueued only once the write is durable. What changed is who keeps the ordering for shape 2. Three of the four command-handler call sites named above contain no SaveChangesAsync of their own: they are MutateEntityHandlerBase subclasses (MMCA.Common/Source/Core/MMCA.Common.Application/UseCases/Crud/MutateEntityHandlerBase.cs:320, over the shared MutateEntityHandlerCore at :52), and the base saves once for every subclass.

  1. The save is one line in the shared base, not one per handler. MutateEntityHandlerCore.MutateCoreAsync awaits attemptUnitOfWork.SaveChangesAsync at MutateEntityHandlerBase.cs:303, calls LogMutated at :305, then awaits the OnMutatedAsync post-save hook at :306. The enqueue is the body of that hook, so "enqueue below the save" is structural rather than remembered: a subclass has no way to express the other order.
    • CloseLivePollHandler declares the base at MMCA.ADC/Source/Modules/Engagement/MMCA.ADC.Engagement.Application/LivePolls/UseCases/Close/CloseLivePollHandler.cs:24, overrides LogMutated at :64-65 and calls EnqueueClosed(poll) from OnMutatedAsync at :73; the queue write is at :94-95.
    • OpenLivePollHandler declares it at .../LivePolls/UseCases/Open/OpenLivePollHandler.cs:26, overrides LogMutated at :79-80 and calls EnqueueOpened(poll) at :88; the queue write is at :109-110.
    • ModerateQuestionHandler declares it at .../SessionQuestions/UseCases/Moderate/ModerateQuestionHandler.cs:28, overrides LogMutated at :81-82, and forwards to EnqueueModeratedAsync(question, command.Action, _wasPending) as the expression body of OnMutatedAsync at :89; that helper wraps its work in BestEffort.ExecuteAsync (:138) with one queue write per branch (:140 the moderation event, :154 the pending count).
  2. One call site still writes the ordering by hand. SubmitQuestionHandler implements ICommandHandler<SubmitQuestionCommand, Result<SessionQuestionDTO>> directly (.../SessionQuestions/UseCases/Submit/SubmitQuestionHandler.cs:31), so it owns its own sequence: it awaits unitOfWork.SaveChangesAsync at :103, logs at :105, and awaits EnqueueSubmittedAsync(question) at :107, whose BestEffort.ExecuteAsync body (:130-131) writes the queue at :142 (approved) and :157 (pending count). This is the one handler where an edit that lifted the enqueue above the save would break the property with nothing to catch it.
  3. Point 3 above is narrowed, not withdrawn. Its premise still holds: TransactionalCommandDecorator wraps only commands implementing ITransactional and passes everything else straight through (MMCA.Common/Source/Core/MMCA.Common.Application/UseCases/Decorators/TransactionalCommandDecorator.cs:28-29, marker at .../UseCases/Markers/ITransactional.cs:6), and no Engagement command implements it, so SaveChangesAsync is the commit for all four sites. What no longer holds is its closing sentence: for the three base-class handlers a pipeline does keep the ordering, and the rule a future edit has to keep by hand lives in SubmitQuestionHandler alone.
← Back to Architecture Decision Records View source on GitHub ↗