Ivan Ball-llovera

↑↓ to navigate Enter to open "…" all these words ANDOR to combine

Architecture Decision Record

ADR-052: Background Job Execution (Bounded Queue plus Hosted Drain)

Status

Superseded by ADR-121 (2026-09-11). The expensive half of this decision (Wait full mode, in-queue dedup, a 409 refusal, the session-scoring queue and drain) moved to durable internal commands (ADR-114), while the ephemeral half continues unchanged in ADR-121; the body below is left in place as history.

Previously: Accepted (2026-07-24). Revised 2026-08-23 (post-commit enqueue is recorded as two patterns, not one) and 2026-08-31 (three of the four command-handler enqueue sites inherit the save-then-enqueue ordering from a shared base class instead of writing it) and 2026-09-19 (the fourth site moved its save into a distributed-lock helper): see the revisions at the end. Revised 2026-10-21: the drain now loops WaitToReadAsync plus TryRead and, on shutdown, drains what is still queued within a 5-second budget, logging what it abandons; the producer API is a void Enqueue rather than TryEnqueue; the session-scoring queue no longer exists in source; and the bookmark cache-eviction signal is a second, coalescing (DropWrite) instance. Revised 2026-10-26: anchors refreshed after the SubmitQuestionHandler and OpenLivePollHandler bodies shifted.

Context

Some requests trigger work that cannot run inside the request: an AI scoring pass over an event's sessions takes minutes and issues one paid API call per session, and a live-channel broadcast must not put a gRPC round trip on the hot path of a vote. The endpoint should accept the request and return, leaving the work to run elsewhere.

The obvious shortcut is to start an untracked task and return, which is what the session-scoring endpoint did (_ = RunScoringInBackgroundAsync(eventId)). A code review found three defects that follow from the shape rather than from that particular implementation:

  • The host does not know the work exists. IHostApplicationLifetime cannot wait for it and shutdown cannot cancel it, so an Azure Container Apps scale-in or a deploy tears it down mid-run with nothing recorded. For a multi-minute paid pass, that is spend with no result and no trace.
  • Nothing deduplicates it. Two clicks started two concurrent passes over the same event: double the API calls, racing each other's writes, and no way for the caller to learn a run was already in flight.
  • Failure handling drifts. Each fire-and-forget site invents its own try/catch, so the failure posture (swallow, log, retry) is decided per call site instead of once.

There is a second, unrelated need with the same shape: work that must happen after the request's transaction commits and must not add latency to it. The live-channel broadcast is that case, and it already used a bounded queue plus a hosted drain (ADR-039). The two wanted the same mechanism.

This is deliberately not a distributed job system. Every case here is best-effort or re-triggerable work owned by one service; durable, cross-process scheduling is what the outbox (ADR-003) and the broker (ADR-008) are for, and this ADR does not compete with them.

Decision

In-process background work runs as a bounded queue plus a single-reader hosted drain. Nothing starts an untracked Task from a request.

  • A bounded Channel<T> per job kind, registered as a singleton, with the concrete type and its interface both resolving to the one instance (TryAddSingleton<TQueue>() plus TryAddSingleton<IQueue>(sp => sp.GetRequiredService<TQueue>())). Registering them separately would give producers a queue nobody drains.
  • A BackgroundService drain per queue, SingleReader, consuming with a WaitToReadAsync plus TryRead loop on the stopping token. Because it is a hosted service the host owns the work: shutdown completes the queue, cancels the loop, and drains what is still queued within a bounded budget, logging whatever it abandons. The drain resolves scoped services through IServiceScopeFactory per item, since it is itself a singleton.
  • The full mode encodes what the work is worth.
    • Ephemeral work uses BoundedChannelFullMode.DropOldest: under backpressure the freshest broadcast is worth more than the oldest and the request path must never block. Until the queue is completed at shutdown, TryWrite then returns true (it evicts to make room), so a caller cannot learn from its return value that anything was dropped; the channel's itemDropped callback is the only real signal and must be wired to a counter and a log. After completion a write is refused silently.
    • Coalescing work uses BoundedChannelFullMode.DropWrite on a capacity of one: a request made while one is already pending is absorbed into it, because the pending item already covers it.
    • Expensive work no longer uses this pattern. The Wait mode, in-queue dedup by natural key and the 409 refusal for a duplicate were retired with the session-scoring queue; scoring runs as a durable internal command (ADR-114) that takes a distributed claim per event and skips a run another replica already holds.
  • One failure posture per drain, stated once. The drain catches per item so one failed run cannot kill the loop, and handles shutdown cancellation separately from failure so a graceful restart is not recorded as an error.
  • Post-commit work is enqueued after the write is durable, never beside it. The failure this rules out is enqueuing while the write can still be undone, which leaves the queued work describing state that never persisted. Two shapes satisfy it and both are in use:
    • From a domain event handler, which gets post-commit delivery from the existing deferral (ADR-003) with no sequencing code in the command handler: domain-event dispatch inside a transactional command is deferred until after the commit succeeds and dropped on rollback.
    • From the command handler side of a non-transactional command, once its save has returned, for commands that do not opt into the ADR-014 transaction decorator (which wraps only commands implementing ITransactional, so for the rest SaveChangesAsync is the commit). Most of these handlers do not write the ordering themselves: a MutateEntityHandlerBase subclass has no save of its own, and the shared base saves and then calls the OnMutatedAsync post-save hook the handler overrides to enqueue, so the order is structural. A handler that saves inline instead keeps the ordering as statement order, where the enqueue call has to stay below the save.

Two implementations exist: LiveChannelPublishQueue / LiveChannelPublishProcessor (ephemeral, DropOldest, ADR-039) and BookmarkCacheEvictionSignal / BookmarkCacheEvictionProcessor (coalescing, DropWrite, capacity one, a paced drain of Conference cache-eviction broadcasts). The expensive instance (SessionScoringQueue / SessionScoringProcessor, Wait, dedup by event id) no longer exists in source; session scoring runs as a durable internal command (ADR-114).

Rationale

  • The host lifetime is the point. A BackgroundService is the only in-process shape the host can cancel and await. Everything else in the decision follows from wanting that.
  • The queue is where the policy lives. Capacity and full mode are properties of the work, and putting them in the queue type means a caller cannot get them wrong: it calls Enqueue, which returns nothing, and the queue applies capacity and full mode on its behalf.
  • Single reader gives ordering for free, which the live-channel case needs per session. Paid work, which wanted refusal rather than silent coalescing and a serialized run, left this pattern for durable internal commands (ADR-114).

Trade-offs

  • In-process only. The queue does not survive a restart and does not span replicas. Accepted because every current job is either ephemeral (a lost broadcast is a missed UI refresh) or re-triggerable (an organizer can click score again). Work that must survive a crash belongs in the outbox (ADR-003), not here.
  • Dedup is per replica. Two replicas can each accept a run for the same event. Today the scoring endpoint is organizer-only and low-traffic, so the exposure is small; making it cluster-wide would need a distributed lock or a claim row, which is the point at which this should become a real job system instead.
  • Backpressure is felt differently by the two modes, and choosing the wrong one is a silent bug in either direction: DropOldest on expensive work discards paid runs, and Wait on ephemeral work turns a slow peer into refused broadcasts. The mode is a per-job decision, not a default.
  • A drain is a serialization point. One reader means a slow item delays the queue behind it. That is wanted for scoring and harmless for broadcasts at the observed conference-day load; a job kind that needs parallelism needs its own queue rather than a wider reader, or ordering is lost.

ADR-003 (the outbox, for work that must be durable and cross-process, and the post-commit deferral this relies on), ADR-008 (the broker, for cross-service work), ADR-014 (the transactional decorator whose commit boundary post-commit work attaches to), ADR-039 (live channel push, the ephemeral instance of this pattern), ADR-025 (startup warm-up, the other hosted-service use in the framework).

Revision (2026-08-23)

The decision is unchanged: post-commit work is still enqueued only once the write is durable. What changed is the record of how. This ADR stated the domain-event handler as the mechanism; the live-channel enqueue call sites split two-to-four the other way, so the decision bullet now names both shapes.

  1. Two call sites enqueue from an IDomainEventHandler. LivePollVoteChangedHandler implements IDomainEventHandler<LivePollVoteChanged> (MMCA.ADC/Source/Modules/Engagement/MMCA.ADC.Engagement.Application/LivePolls/DomainEventHandlers/LivePollVoteChangedHandler.cs:41) and enqueues the poll.results-changed broadcast at :79; SessionQuestionUpvoteChangedHandler implements IDomainEventHandler<SessionQuestionUpvoteChanged> (.../SessionQuestions/DomainEventHandlers/SessionQuestionUpvoteChangedHandler.cs:42) and enqueues the upvote-count broadcast at :80. Both are singletons that open their own scope (:53 and :54 respectively) and wrap the work in BestEffort.ExecuteAsync (:51, :52).
  2. Four call sites enqueue from the command handler, on the line after the save. There is no domain event in between. (Superseded by the Revision (2026-08-31): three of these four handlers hold no save of their own today, and their anchors below have moved. The text is left in place as the record of what was true when written.)
    • CloseLivePollHandler awaits unitOfWork.SaveChangesAsync at .../LivePolls/UseCases/Close/CloseLivePollHandler.cs:70, logs at :72 and calls EnqueueClosed(poll) at :74; the queue write itself is at :95.
    • OpenLivePollHandler saves at .../LivePolls/UseCases/Open/OpenLivePollHandler.cs:85 and calls EnqueueOpened(poll) at :89, queue write at :110.
    • SubmitQuestionHandler saves at .../SessionQuestions/UseCases/Submit/SubmitQuestionHandler.cs:103 and awaits EnqueueSubmittedAsync(question) at :107; that helper is a BestEffort.ExecuteAsync body (:130-131) with one queue write per branch (:142 approved, :157 pending-count).
    • ModerateQuestionHandler saves at .../SessionQuestions/UseCases/Moderate/ModerateQuestionHandler.cs:80 and awaits EnqueueModeratedAsync(...) at :84, queue writes at :138 and :152.
  3. The safety property still holds for those four, because they are not transactional commands. TransactionalCommandDecorator wraps only commands implementing the ITransactional marker and passes everything else straight through (MMCA.Common/Source/Core/MMCA.Common.Application/UseCases/Decorators/TransactionalCommandDecorator.cs:28-29, marker at .../UseCases/Markers/ITransactional.cs:6). No Engagement command implements it, so for these handlers SaveChangesAsync is the commit and an enqueue below it is genuinely post-commit. The difference from shape 1 is where the ordering lives: in the command handler's statement order rather than in the ADR-003 deferral, so it is a rule a future edit to those handlers has to keep rather than one the pipeline keeps for them. (The last sentence is superseded by the Revision (2026-08-31): it describes one of the four handlers, not all four.)

Revision (2026-08-31)

The decision and the safety property are unchanged: post-commit work is still enqueued only once the write is durable. What changed is who keeps the ordering for shape 2. Three of the four command-handler call sites named above contain no SaveChangesAsync of their own: they are MutateEntityHandlerBase subclasses (MMCA.Common/Source/Core/MMCA.Common.Application/UseCases/Crud/MutateEntityHandlerBase.cs:320, over the shared MutateEntityHandlerCore at :52), and the base saves once for every subclass.

  1. The save is one line in the shared base, not one per handler. MutateEntityHandlerCore.MutateCoreAsync awaits attemptUnitOfWork.SaveChangesAsync at MutateEntityHandlerBase.cs:303, calls LogMutated at :305, then awaits the OnMutatedAsync post-save hook at :306. The enqueue is the body of that hook, so "enqueue below the save" is structural rather than remembered: a subclass has no way to express the other order.
    • CloseLivePollHandler declares the base at MMCA.ADC/Source/Modules/Engagement/MMCA.ADC.Engagement.Application/LivePolls/UseCases/Close/CloseLivePollHandler.cs:24, overrides LogMutated at :64-65 and calls EnqueueClosed(poll) from OnMutatedAsync at :73; the queue write is at :94-95.
    • OpenLivePollHandler declares it at .../LivePolls/UseCases/Open/OpenLivePollHandler.cs:26, overrides LogMutated at :79-80 and calls EnqueueOpened(poll) at :88; the queue write is at :109-110.
    • ModerateQuestionHandler declares it at .../SessionQuestions/UseCases/Moderate/ModerateQuestionHandler.cs:28, overrides LogMutated at :81-82, and forwards to EnqueueModeratedAsync(question, command.Action, _wasPending) as the expression body of OnMutatedAsync at :89; that helper wraps its work in BestEffort.ExecuteAsync (:138) with one queue write per branch (:140 the moderation event, :154 the pending count).
  2. One call site still writes the ordering by hand. SubmitQuestionHandler implements ICommandHandler<SubmitQuestionCommand, Result<SessionQuestionDTO>> directly (.../SessionQuestions/UseCases/Submit/SubmitQuestionHandler.cs:31), so it owns its own sequence: it awaits unitOfWork.SaveChangesAsync at :103, logs at :105, and awaits EnqueueSubmittedAsync(question) at :107, whose BestEffort.ExecuteAsync body (:130-131) writes the queue at :142 (approved) and :157 (pending count). This is the one handler where an edit that lifted the enqueue above the save would break the property with nothing to catch it. (Superseded by the Revision (2026-09-19): the save named here is no longer a statement in HandleAsync. The text is left in place as the record of what was true when written.)
  3. Point 3 above is narrowed, not withdrawn. Its premise still holds: TransactionalCommandDecorator wraps only commands implementing ITransactional and passes everything else straight through (MMCA.Common/Source/Core/MMCA.Common.Application/UseCases/Decorators/TransactionalCommandDecorator.cs:28-29, marker at .../UseCases/Markers/ITransactional.cs:6), and no Engagement command implements it, so SaveChangesAsync is the commit for all four sites. What no longer holds is its closing sentence: for the three base-class handlers a pipeline does keep the ordering, and the rule a future edit has to keep by hand lives in SubmitQuestionHandler alone.

Revision (2026-09-19)

The decision and the safety property are unchanged: post-commit work is still enqueued only once the write is durable. What changed is where the save sits in the one remaining hand-written site. The per-user open-question cap in SubmitQuestionHandler did not hold under a burst from one account, because the count read and the insert are not one atomic statement, so the count, the creation and the save moved together into a private helper that runs under a cross-replica claim on (session, user) (ADR-108).

  1. SubmitQuestionHandler no longer saves inline in HandleAsync. It takes IDistributedLock as a constructor parameter (MMCA.ADC/Source/Modules/Engagement/MMCA.ADC.Engagement.Application/SessionQuestions/UseCases/Submit/SubmitQuestionHandler.cs:34, interface at MMCA.Common/Source/Core/MMCA.Common.Application/Interfaces/IDistributedLock.cs:30) and calls CreateUnderClaimAsync at :91. That helper (:142) acquires the claim with TryAcquireAsync at :148 on the key built by ClaimKey at :107, answers a refused claim with the same cap error as a refused count (:151-154), counts the open questions at :159, creates and adds the question at :172 and :182, and awaits unitOfWork.SaveChangesAsync at :184.
  2. The ordering is still written by hand, one level up. HandleAsync logs at :97 and awaits EnqueueSubmittedAsync(question) at :99, below the CreateUnderClaimAsync call that now contains the save; the enqueue helper's BestEffort.ExecuteAsync body (:205-206) writes the queue at :217-218 (approved) and :232-233 (pending count). Point 2 of the Revision (2026-08-31) holds in substance: this is still the one site where the property lives in statement order rather than in a base class, and the statement the enqueue has to stay below is now the helper call.
  3. The claim is scoped so the post-commit rule survives it. It is taken after the gRPC live-window lookup, so no lock is held across a cross-service call, and it is released when the helper's handle is disposed, before anything is enqueued (:126-132 state both), with a 30-second time to live (:50) as the crash backstop. With no Redis configured the lock falls back to its in-process implementation, so exclusion is per replica again (:134-140), which is the same honest limit the "dedup is per replica" trade-off above states for in-process dedup.

Revision (2026-10-21)

Re-verified against current source. The ephemeral half and the post-commit rule are unchanged: the live-channel queue is still a bounded DropOldest channel with a single reader, drained by one hosted service that opens a scope per item, and every enqueue still sits below the save. What moved is how the drain consumes and stops, the producer API, and the expensive instance, which is gone.

  1. The drain no longer uses ReadAllAsync. LiveChannelPublishProcessor.ExecuteAsync loops WaitToReadAsync plus TryRead, checking the stopping token before every dequeue (MMCA.ADC/Source/Modules/Engagement/MMCA.ADC.Engagement.Infrastructure/Live/LiveChannelPublishProcessor.cs:89-97), and still catches shutdown cancellation apart from failure (:100). A StopAsync override (:53-82) completes the queue and cancels the worker, which aborts and loses the in-flight publish (:62-66); only if the worker has returned within ShutdownDrainBudget (5 seconds, :50) does it drain the remaining items (:72), and it logs the count it abandons (:77-80). The per-item scope is at :130. The Decision bullet now states this.
  2. The producer API is void Enqueue, not TryEnqueue. LiveChannelPublishQueue.Enqueue discards the TryWrite result (MMCA.ADC/Source/Modules/Engagement/MMCA.ADC.Engagement.Application/Live/LiveChannelPublishQueue.cs:56-60); Complete (:66) closes the writer at shutdown, and the channel is DropOldest with SingleReader (:36-37). The Rationale bullet now states this.
  3. The session-scoring queue and drain no longer exist. Neither SessionScoringQueue nor SessionScoringProcessor appears under MMCA.ADC/Source, and no channel there uses BoundedChannelFullMode.Wait. Scoring runs as a durable internal command (ADR-114), as the Status states. The scoring handler takes a distributed claim per event and skips a run another replica holds (MMCA.ADC/Source/Modules/Conference/MMCA.ADC.Conference.Application/Sessions/UseCases/DecisionSupport/ScoreEventSessions/ScoreEventSessionsInternalCommandHandler.cs:84, log message at :132), so the Decision and Rationale no longer state Wait, in-queue dedup or a 409 refusal as live policy.
  4. A second instance of the pattern exists, with a third full mode. BookmarkCacheEvictionSignal is a bounded channel of capacity one with DropWrite and SingleReader (MMCA.ADC/Source/Modules/Engagement/MMCA.ADC.Engagement.Application/UserSessionBookmarks/Services/BookmarkCacheEvictionSignal.cs:27-33), registered as a singleton (MMCA.ADC/Source/Modules/Engagement/MMCA.ADC.Engagement.Application/DependencyInjection.cs:58) and drained by the hosted BookmarkCacheEvictionProcessor (MMCA.ADC/Source/Modules/Engagement/MMCA.ADC.Engagement.Infrastructure/Caching/BookmarkCacheEvictionProcessor.cs:35, WaitToReadAsync plus TryRead at :56-60; AddHostedService at MMCA.ADC/Source/Modules/Engagement/MMCA.ADC.Engagement.Infrastructure/DependencyInjection.cs:24). The Decision now names two implementations and the coalescing full mode, and the TryWrite note is qualified by Complete, after which a live-channel write is refused (LiveChannelPublishQueue.cs:51-54, :66).
  5. SubmitQuestionHandler now waits briefly for the claim. TryAcquireAsync receives ClaimWait (2 seconds, MMCA.ADC/Source/Modules/Engagement/MMCA.ADC.Engagement.Application/SessionQuestions/UseCases/Submit/SubmitQuestionHandler.cs:58) and only then refuses. The substance of the Revision (2026-09-19) holds: the save is inside the claim helper and the enqueue follows it.
  6. MutateEntityHandlerCore has a no-op path. When the mutation reports SaveSkipped (MMCA.Common/Source/Core/MMCA.Common.Application/UseCases/Crud/MutateEntityHandlerBase.cs:310), neither the save nor the post-save hooks run; otherwise save-then-hook ordering is still structural.
  7. Anchors re-verified against current source (the dated revisions above keep their original anchors as written):
    • TransactionalCommandDecorator.cs: passthrough check at :30.
    • MutateEntityHandlerBase.cs: MutateEntityHandlerCore at :52, SaveChangesAsync at :322, LogMutated at :324, OnMutatedAsync at :325, MutateEntityHandlerBase at :339 (DTO variant at :362).
    • SubmitQuestionHandler.cs: class at :32, IDistributedLock parameter at :35, ICommandHandler base at :39, ClaimTimeToLive at :50, CreateUnderClaimAsync call at :97 and definition at :148, log at :103, EnqueueSubmittedAsync call at :105, ClaimKey at :113, TryAcquireAsync at :154, SaveChangesAsync at :190, BestEffort body at :211-212, queue writes at :223 and :238.
    • SessionQuestionUpvoteChangedHandler.cs: IDomainEventHandler at :44, BestEffort at :54, scope at :56, enqueue at :87.

Revision (2026-10-26)

Re-verified against current source. Nothing in the decision, the post-commit rule or the ephemeral half moved: SubmitQuestionHandler still saves inside the claim helper and enqueues below that call, and OpenLivePollHandler still enqueues from the OnMutatedAsync post-save hook of MutateEntityHandlerBase. Only line anchors moved: the SubmitQuestionHandler body after the ClaimWait field sits a uniform four lines higher than item 7 of the Revision (2026-10-21) records, and OpenLivePollHandler sits one line higher than the Revision (2026-08-31) records. The dated revisions above keep their anchors as written.

  1. Anchors re-verified against current source:
    • SubmitQuestionHandler.cs (MMCA.ADC/Source/Modules/Engagement/MMCA.ADC.Engagement.Application/SessionQuestions/UseCases/Submit/SubmitQuestionHandler.cs): class at :32, IDistributedLock parameter at :35, ICommandHandler base at :39, ClaimTimeToLive at :50, ClaimWait at :58, CreateUnderClaimAsync call at :95 and definition at :144, LogQuestionSubmitted at :99, EnqueueSubmittedAsync call at :101, ClaimKey at :109, TryAcquireAsync at :150, SaveChangesAsync at :186, EnqueueSubmittedAsync and its BestEffort.ExecuteAsync body at :207-208, queue writes at :219 (approved) and :234 (pending count).
    • OpenLivePollHandler.cs (MMCA.ADC/Source/Modules/Engagement/MMCA.ADC.Engagement.Application/LivePolls/UseCases/Open/OpenLivePollHandler.cs): MutateEntityHandlerBase at :26, LogMutated override at :78, EnqueueOpened(poll) call at :87, queue write at :108.
    • MutateEntityHandlerBase.cs: MutateEntityHandlerCore at :52, SaveSkipped check at :310, SaveChangesAsync at :322, LogMutated at :324, OnMutatedAsync at :325, MutateEntityHandlerBase at :339 (DTO variant at :362), unchanged.
    • SessionQuestionUpvoteChangedHandler.cs: IDomainEventHandler at :44, BestEffort at :54, enqueue at :87, unchanged.
← Back to Architecture Decision Records View source on GitHub ↗